Cloning Oracle Applications Release 12 with Rapid Clone [ID 406982.1]
________________________________________
Cloning Oracle Applications Release 12 with Rapid Clone
This knowledge document describes the process of cloning an Oracle Applications Release 12 system.
The content applies to all Release 12.x.x versions, such as 12.0, 12.0.4, and 12.1.x. Where applicable, 12.0.x releases will in general be referred to as Release 12.0, and 12.1.x releases as Release 12.1.
The most current version of this document can be obtained in Oracle Support Knowledge Document 406982.1.
There is a change log at the end of this document.
In This Document
• Section 1: Prerequisite Tasks
• Section 2: Cloning Tasks
• Section 3: Finishing Tasks
• Section 4: Advanced Cloning Options
Terminology
Cloning is the process used to create a copy of an existing Oracle Applications system. There are various scenarios for cloning an Oracle Applications system, including:
• Standard cloning - Making a copy of an existing Oracle Applications system, for example a copy of a production system to test updates.
• System scale-up - Adding new machines to an Oracle Applications system to provide the capacity for processing an increased workload.
• System transformations - Altering system data or file systems, including actions such as platform migration, data scrambling, and provisioning of high availability architectures.
• Patching and upgrading - Delivering new versions of Applications components, and providing a mechanism for creating rolling environments to minimize downtime.
An important principle in Oracle Applications cloning is that the system is cloned, rather than the topology. Producing an exact copy of the patch level and data is much more important than creating an exact copy of the topology, as a cloned system must be able to provide the same output to the end user as the source system. However, while a cloned system need not have the full topology of its source, it must have available to it all the topology components that are available to the source.
Note: The Cloning methodology posted on this document requires full Autoconfig compliance.
Conventions used in this document include the following:
Term or Usage Meaning or Action
Source system Oracle Applications system being cloned.
Target system Oracle Applications system being created as a copy of the source.
APPLMGR User that owns the application tier file system (APPL_TOP and application tier technology stack).
ORACLE User that owns the database tier file system (RDBMS ORACLE_HOME and database files).
CONTEXT_NAME The CONTEXT_NAME variable refers to the name of the Applications context file. By default, CONTEXT_NAME is [SID]_[HOSTNAME].
Monospace Text Represents command line text. Type this command exactly as shown.
[ ] Text enclosed in brackets represents a variable. Substitute a value for the variable text. Do not type the brackets.
Section 1: Prerequisite Tasks
Before cloning, prepare the source system by applying any required patches and running AutoConfig.
1. Verify OS requirements on target system
Before cloning to a new server, ensure the target system meets all the requirements for Oracle Applications Release 12 stated on the Oracle Applications Release Notes, and on the Oracle Applications Installation and Upgrade Notes for each Platform. For the latest installation guidelines refer to My Oracle Support Knowledge Document 405565.1.
Note: On Microsoft Windows, Rapid Clone is not currently certified for use from Domain User Accounts.
2. Verify source and target system software components and versions
In addition to the Oracle Applications software requirements (see Oracle Applications Installation Guide: Using Rapid Install), the following software component versions must exist on the source or target nodes as applicable. The 'Location' column indicates the node where the software component must reside.
Table 1: Software Requirements
Software Component Minimum Version Required Location Comments
Zip 2.3
(or higher) All source system nodes Download from InfoZip. Zip must be in your $PATH. If using files bigger than 2Gb, you should use InfoZip ZIP 3.0 or higher.
Unzip 5.52
(or higher) All source system nodes Download from InfoZip. Unzip must be in your $PATH. If using files bigger than 2Gb, you should use InfoZip UNZIP 5.52 or higher.
Operating system utilities N/A All target system nodes The required operating system utilities for your platform must be in your $PATH when running adcfgclone.pl. For example, make, ld, and ar on UNIX. Refer to Oracle Applications Installation Guide: Using Rapid Install(see Footnote 1)
Perl 5.x All target system nodes Use the Perl shipped with OracleAS 10.1.3 and Database 10g, or download it from Perl.com. Perl must be in your $PATH, and $PERL5LIB must be set correctly before cloning.
Footnote 1 This is the Release 12.1.1 version; versions for earlier releases are also available
from the Oracle E-Business Suite Online Documentation Library
3. Apply the latest AD patch
Refer to My Oracle Support to obtain the latest AD patch. At the time of writing this note, the following patches were available:
For Release 12.0:
Apply patches as listed in Table 2.a.
Table 2.a: Release 12.0 AD patches
Patch Description
6510214
R12.AD.A.DELTA.4
For Release 12.1:
Apply patches as listed in Table 2.b.
Table 2.b: Release 12.1 AD patches
Patch Description
9239089
R12.AD.B.DELTA.3
4. Apply the latest AutoConfig template patch
Update the Oracle Applications file system with the latest AutoConfig template files by applying the TXK AutoConfig Template Rollup patch to all application tier server nodes. Refer to My Oracle Support Knowledge Document 387859.1 for details of the latest AutoConfig Template Rollup patch.
5. Apply the latest Rapid Clone patches
Update the Oracle Applications file system with the latest Rapid Clone files by applying the following patches to all Applications nodes.
o For Release 12.0:
Apply patches as listed in Table 3.a.
Table 3.a: Release 12.0 Rapid Clone patches
Patch Description
5484000
Oracle E-Business Suite 12.0.2 Release Update Pack (RUP2) or higher
9171651:R12.OAM.A
12.0 RAPIDCLONE CONSOLIDATED FIXES JUL/2010
9833058:R12.OAM.A
HOT CLONE FAILS WITH ORA-00201 DURING RECOVERY MANAGER
o For Release 12.1:
Apply patches as listed in Table 3.b.
Table 3.b: Release 12.1 Rapid Clone patches
Patch Description
9171651:R12.OAM.B
12.1 RAPIDCLONE CONSOLIDATED FIXES JUL/2010
9833058:R12.OAM.B
HOT CLONE FAILS WITH ORA-00201 DURING RECOVERY MANAGER
12404574:R12.OAM.B
ORACLE_HOME REGISTRATION DOES NOT HAPPEN WITH CENTRAL INVENTORY ON LOZ
12598630:R12.OAM.B
R12.1 ONE-OFF FOR S_DB_LISTENER BUG 12362010
o Other Patches (All releases):
Apply patches as listed in Table 3.c.
Table 3.c: Other Patches
Patch Description
8246709
Required for Microsoft Windows if using OracleAS 10.1.3.4. This patch must be re-applied to the OracleAS 10.1.3.4 ORACLE_HOME before every cloning operation.
Warning: Failing to use the latest code may jeopardize the success of the clone. If new Rapid Clone or AutoConfig updates are applied to the system, steps 6, 7, and 8 below must be executed again in order to apply the new files to the database node.
6. Run AutoConfig on the application tiers
Follow the steps under section " Run AutoConfig on the Application Tiers " in My Oracle Support Knowledge Document 387859.1 to run AutoConfig on all application tier nodes.
7. Synchronize appsutil on the database tier nodes
Follow the steps under section "Copy AutoConfig to the RDBMS ORACLE_HOME" in My Oracle Support Knowledge Document 387859.1 to copy AutoConfig and Rapid Clone files to each database node via the admkappsutil.pl utility.
8. Run AutoConfig on the database tier
Follow the steps under section "Run AutoConfig on the Database Tier" in My Oracle Support Knowledge Document 387859.1 to run AutoConfig on the database tier nodes.
9. Maintain Snapshot Information
Log in to each application tier node as the APPLMGR user, and run "Maintain Snapshot Information" in AD Administration. To update the snapshot, please select the following options "Update Current View Snapshot" and "Update Complete APPL_TOP".
Note: If a snapshot was never created for that APPL_TOP, you will need to create a new one before proceeding with the clone. Refer to Oracle Applications Maintenance Utilities for more information (this is the Release 12.1 version; versions for earlier releases are also available from theOracle E-Business Suite Online Documentation Library).
Section 2: Cloning Tasks
Use Rapid Clone to create template files for cloning on the source system. After the source system is copied to the target, Rapid Clone updates these templates to contain the new target system configuration settings.
Note: Rapid Clone never changes the source system configuration.
The cloning process consists of three phases, each of which is made up of several logical sections and their steps.
1. Prepare the source system
Execute the following commands to prepare the source system for cloning:
a. Prepare the source system database tier for cloning
Log on to the source system as the ORACLE user, and run the following commands:
$ cd [RDBMS ORACLE_HOME]/appsutil/scripts/[CONTEXT_NAME]
$ perl adpreclone.pl dbTier
b. Prepare the source system application tier for cloning
Log on to the source system as the APPLMGR user, and run the following commands on each node that contains an APPL_TOP:
$ cd [INST_TOP]/admin/scripts
$ perl adpreclone.pl appsTier
Note: If new Rapid Clone or AutoConfig updates are applied to the system, adpreclone.pl must be executed again on the dbTier and on the appsTier in order to apply the new files into the clone directory structures that will be used during the cloning configuration stage.
2. Copy the source system to the target system
Copy the application tier file system from the source Applications system to the target node by executing the following steps in the order listed. Ensure the application tier files copied to the target system are owned by the target APPLMGR user, and that the database node files are owned by the target ORACLE user.
Note: In the copying tasks below, UNIX/Linux users should ensure that the symbolic links (soft links) are preserved when copying. On most UNIX platforms, this can be accomplished with the cp -RH command. Consult the UNIX man page for the cp command to check the parameters available on your platform.
For example: cd /target_dest_dir/db cp -RH /source_dir/db/*
Alternatively, the tar command can be used to compress the directories into a temporary staging area. UNIX/Linux users should ensure that the symbolic links (soft links) are preserved when compressing. On most UNIX platforms, this is the default behavior of tar command. Consult the UNIXman page for the tar command to check the parameters available on your platform
Additionally, verify the permissions of the executables under ORACLE_HOME/bin that can potentially be owned by root (i.e. nmo, nmhs, nmb, etc).
a. Copy the application tier file system
Log on to the source system application tier nodes as the APPLMGR user and shut down the application tier server processes. Copy the following application tier directories from the source node to the target application tier node:
• [APPL_TOP]
• [COMMON_TOP]
• Applications Technology Stack:
[OracleAS Tools ORACLE_HOME]
[OracleAS Web IAS_ORACLE_HOME]
b. Copy the database node file system
Log on to the source system database node as the ORACLE user, and then:
1. Perform a normal shutdown of the source system database
2. Copy the database (.dbf) files from the source system to the target system
3. Copy the source database ORACLE_HOME to the target system
4. Start the source Applications system database and application tier processes
Configure the target system
Run the following commands to configure the target system. You will be prompted for specific target system values such as SID, paths, and ports.
Configure the target system database server
Log on to the target system as the ORACLE user and enter the following commands:
$ cd [RDBMS ORACLE_HOME]/appsutil/clone/bin
$ perl adcfgclone.pl dbTier
Configure the target system application tier server nodes
Log on to the target system as the APPLMGR user and enter the following commands:
$ cd [COMMON_TOP]/clone/bin
$ perl adcfgclone.pl appsTier
Note: If you are cloning between different versions of the Operating System (e.g. Linux 5 to Linux 4) it is recommended that you relink the AD_TOP product binaries using [adrelinknew "ad all"] followed by APPL_TOP executables with adadmin.
Section 3: Finishing Tasks
This section lists tasks that may be necessary, depending on your implementation and the intended use of the cloned system.
1. Update profile options
Rapid Clone updates only site level profile options. If any other profile options are set to instance specific values, you must update them manually.
2. Update printer settings
If the new cloned system needs to utilize different printers, update the target system with the new printer settings now.
3. Update Workflow configuration settings
Cloning an Oracle Applications instance will not update the host and instance-specific information used by Oracle Workflow. Review the tables and columns listed in Table 4 to check for any instance-specific data in the Workflow configuration on the target system.
Table 4: Workflow configuration settings
Table Name Column Name Column Value Details
WF_NOTIFICATION_ATTRIBUTES TEXT_VALUE Value starts with http://[old web host]: Update to new web host.
WF_ITEM_ATTRIBUTE_VALUES TEXT_VALUE Value starts with "http://[old web host]: Update to new web host.
WF_SYSTEMS GUID Using the Workflow Administrator Web Applications responsibility, create a new system defined as the new global database name.
WF_SYSTEMS NAME Replace value with the database global name.
WF_AGENTS ADDRESS Update database link with the new database global name.
FND_FORM_FUNCTIONS WEB_HOST_NAME Update with the new web host name.
FND_FORM_FUNCTIONS WEB_AGENT_NAME Update to point at the new PL/SQL listener name.
FND_CONCURRENT_REQUESTS LOGFILE_NAME Update with the correct path to the logfile directory.
FND_CONCURRENT_REQUESTS OUTFILE_NAME Update with the new directory path on the target system.
4. Verify the APPLCSF variable setting
Source the APPS environment and review that the variable APPLCSF (identifying the top-level directory for concurrent manager log and output files) points to a suitable directory. To modify it, change the value of the s_applcsf variable in the context file and then run AutoConfig.
5. Update the SESSION_COOKIE_DOMAIN value in ICX_PARAMETERS
If the target system is in a different domain name than the source system and SESSION_COOKIE_DOMAIN was not null in the source system, update that value to reflect the new domain name.
6. Re-Implement SSL and SSO configuration
If the Source System was SSL or SSO enabled, and the Target is wished to be SSL or SSO enabled, then reconfigure the Target by following the SSL/SSO documentation. Otherwise, if the Target is wished to be non-SSL or non-SSO, then follow the same SSL/SSO documentation to undo the SSL/SSO setup.
Section 4: Advanced Cloning Options
This section describes various advanced cloning procedures that may need to be employed in the appropriate circumstances.
Option 1: Refreshing a Target System
You may need to refresh the target system periodically to synchronize it with changes to the source system.
Note: Back up the target context file on the target system before refreshing the database or application tiers.
To refresh the target system, perform the following steps as described in previous sections:
1. Prepare the source system.
2. Copy the source system to the target system.
a. If the application tier file system if the APPL_TOP, COMMON_TOP, or technology stack needs to be refreshed, copy the portion of the application tier file system that has been updated
b. If the RDBMS ORACLE_HOME or the database needs to be refreshed, copy the database node file system. If refreshing the database, the ORACLE_HOME should be refreshed at the same time.
3. Configure the target system.
Specify the existing target system context file when running adcfgclone.pl commands.
a. Configure the target system database server by logging on to the target system as the ORACLE user and entering the following commands to configure and start the database:
$cd [RDBMS ORACLE_HOME]/appsutil/clone/bin
perl adcfgclone.pl dbTier [Database target context file]
Where Database target context file is [RDBMS ORACLE_HOME]/appsutil/[Target CONTEXT_NAME].xml
b. Configure the target system application tier server nodes by logging on to the target system as the APPLMGR user and entering the following commands:
$ cd [COMMON_TOP]/clone/bin
$ perl adcfgclone.pl appsTier [APPL_TOP target context file]
Where APPL_TOP target context file is [INST_TOP]/appl/admin/[Target CONTEXT_NAME].xml
4. Perform the standard finishing tasks.
Option 2: Cloning a Multi-Node System
This procedure allows the source system or target system to be a multi-node system. As of Release 12, all APPL_TOPs are unified APPL_TOPs. This means that all files required for all application tier services are installed on every application tier node. Thus, only one copy of the applications tier node files needs to be copied to the target system, regardless of whether a shared file system is being used on the source or target system. Multiple application tier nodes are distinguished from each other by the services running.
1. Perform the standard prerequisite tasks.
Carry out these steps on all source and target nodes.
2. Carry out the previously-described cloning tasks.
Prepare, copy and configure the cloned Applications system. When creating more than one application tier node on the target system, follow these steps:
1. Perform a full clone (Prepare, copy and configure steps) of the database node and primary application tier node.
2. To add shared application tier nodes on the target system, follow the instructions in My Oracle Support Knowledge Document 384248.1, Section 4: Adding a node to a Shared Application Tier File System.
3. To add non-shared application tier nodes, execute the copy and configure steps as on the primary node.
4. Specify the services to start on each target Applications tier node when responding to the prompts during the configuration step.
3. Perform the required finishing tasks
Option 3: Adding a New Node to an Existing System
You can use Rapid Clone to clone a node and add it to the existing Applications system, a process also known as scale up or scale out. The new node can run the same services as the source node, or different services. Follow the instructions in the Application tier part of Cloning Tasks.
1. Prepare the source system, copy it to the new node and configure it.
2. After adcfgclone.pl completes, source the Applications environment and run the following commands on the target system:
$ cd [COMMON_TOP]/clone/bin
$ perl adaddnode.pl
Note: After adding new nodes, refer to My Oracle Support Knowledge Document 380489.1 for details of how to set up load balancing.
Note: If SQL*Net Access security is enabled in the existing system, you first need to authorize the new node to access the database through SQL*Net. See the Oracle Applications Manager on line help for instructions on how to accomplish this.
Option 4: Cloning an Oracle RAC system
For instructions on how to Clone RAC-Enabled Systems with Rapid Clone, refer to My Oracle Support Knowledge Document 559518.1.
Option 5: Adding a Node to an Existing Oracle RAC Cluster
From Release 12, Rapid Clone is no longer used to migrate a database tier to Oracle RAC. Refer to My Oracle Support Knowledge Document 388577.1 for instructions on how to perform this task.
Option 6: Cloning the Database Separately
Some situations require the database to be recreated separately, without using Rapid Clone. Typical scenarios are when system downtime is not feasible, or advanced database replication tools like RMAN are being used to copy the database in hot backup mode.
This section documents the steps needed to allow manual creation of the target database control files within the Rapid Clone process. This method needs to be used for databases located on raw partitions, or when cloning a hot backup. Follow the complete steps in Cloning Tasks, but replace Step 3a (Configure the target system database server) with the following steps:
1. Log on to the target system as the ORACLE user
2. Configure the [RDBMS ORACLE_HOME]
$ cd [RDBMS ORACLE_HOME]/appsutil/clone/bin
$ perl adcfgclone.pl dbTechStack
3. Create the target database control files manually
In this step, you copy and recreate the database using your preferred method, such as RMAN restore, Flash Copy, Snap View, or Mirror View.
4. Start the target database in open mode
5. Run the library update script against the database
$ cd [RDBMS ORACLE_HOME]/appsutil/install/[CONTEXT NAME]
$ sqlplus "/ as sysdba" @adupdlib.sql [libext]
Where [libext] should be set to 'sl' for HP-UX, 'so' for any other UNIX platform, or 'dll' for Windows.
6. Configure the target database
The database must be running and open before performing this step.
$ cd [RDBMS ORACLE_HOME]/appsutil/clone/bin
$ perl adcfgclone.pl dbconfig [Database target context file]
Where Database target context file is: [RDBMS ORACLE_HOME]/appsutil/[Target CONTEXT_NAME].xml.
Note: The dbconfig option will configure the database with the required settings for the new target, but it will not recreate the control files.
Change Log
Date Description
Nov 17, 2011 • Removed duplicated patch.
• Added Patch 12598630:R12.OAM.B.
• Added relink recommendation when cloning between slightly different OS versions.
May 16, 2011 • Further clarified SSL/SSO considerations in Section 3, Step 6.
• Further clarified tar considerations in Section 2, Step 2.
Apr 29, 2011 • Added one-off Patch 12404574:R12.OAM.B.
Mar 30, 2011 • Added a note for Autoconfig requirement.
• Added note for patch reinforcement.
Jan 03, 2011 • Corrected URLs for updates.oracle.com
Dec 29, 2010 • Added Patch 9833058 for 12.0 and 12.1 on Section 1, Step 5
• Clarified AD Patches for 12.0 and 12.1
Sep 24, 2010 • Replaced RC Consolidated patch with 9171651
Apr 24, 2010 • Added Patch 9462233 for 12.0 and 12.1 on Section 1, Step 5
Feb 15, 2010 • Addressed general formatting issues
• Added clarifications for versions of ZIP and UNZIP in Section 1
• Added additional step for SSL/SSO in Section 3
• Replaced Rapid Clone Consolidated Patch with 8360494 (JAN/2010)
• Removed other patches since they are now included in 8360494
Jun 24, 2009 • Restructured document.
Jun 23, 2009 • Moved Appendix A into the Advanced Cloning Options and added some more details.
May 08, 2009 • Replaced Rapid Clone Consolidated Patch with 7164226 (May 2009).
Apr 09, 2009 • Corrected hyperlink in Section 4, Step 3.
Mar 13, 2009 • Added Reference to Patch 8246709 for OracleAS 10.1.3.4 Oracle Home.
Mar 02, 2009 • Added Reference to Note 783188.1.
Feb 16, 2009 • Added Clarification for MS Windows (32-bit) on Patch 8221919.
Feb 13, 2009 • Added Patch 8221919 on Section 1, Step 5.
Jan 22, 2009 • Added Patch 7699109 on Section 1, Step 5.
• Added Clarification for RUP2 or higher.
Aug 07, 2008 • Replaced Rapid Clone Consolidated Patch with 6776948 (Jul 2008).
• Removed Patch 6854328 since its included on 6776948 (Jul 2008).
Jul 29, 2008 • Removed Rapid Clone restriction for RAC-enabled Systems in Section 4, Steps 4 and 5.
• Added reference to Document 559518.1 for RAC cloning instructions.
• Added tracking bug number 7276544 to add a node on an existing RAC system without cloning.
• Added tracking bug number 7275583 to add a node while cloning a RAC system.
Mar 24, 2008 • Added Unzip 5.52 to Section 1, Step 2.
• Included Patch 6854328 for Windows 64-bit in Section 5, Step 5.
• Fixed post patch steps on note box.
Feb 28, 2008 • Adjusted heading format in Section 2, Step 2.
Jan 25, 2008 • Added AD.A.DELTA.4 and latest Rapid Clone consolidated patch.
Dec 18, 2007 • Minor layout changes.
• Added requirement to apply the latest AD Patch.
• Added clarification to backup the context file when refreshing a target system on 4.1.
• Included step to verify OS requirements.
Dec 17, 2007 • Modified cp command usage.
Dec 07, 2007 • Included Windows Itanium patch.
Jul 30, 2007 • Removed Windows desupport notice.
• Added Warning to re-run post-clone steps after patching the system.
Jun 25, 2007 • Included clarification on how to use cp command to preserve symbolic links.
May 18, 2007 • Removed reference to HP OpenVMS Alpha since this is not a certified or supported platform for any tier with Release 12.
May 10, 2007 • Corrected link to Oracle Applications Maintenance Utilities on Section 1, Step 7.
Feb 23, 2007 • Added reference to Note 380489.1 for Load Balancing.
Feb 22, 2007 • Added note that using Rapid Clone on Windows is not currently supported.
Jan 24, 2007 • Created initial document.
My Oracle Support Knowledge Document 406982.1 by Oracle E-Business Suite Development
Copyright 2007, 2009, 2010, 2011 Oracle and/or its affiliates. All rights reserved.
Related
________________________________________
Products
________________________________________
• Oracle E-Business Suite > Applications Technology > Lifecycle Management > Oracle Applications Manager
Keywords
________________________________________
$PERL5LIB; ADADDNODE.PL; ADCFGCLONE.PL; ADMKAPPSUTIL.PL; ADPRECLONE.PL; APPL_TOP; AUTOCONFIG; CLONE; CONTEXT; E-BUSINESS; LIBRARY; LOGFILE; MIGRATION; PERMISSION; R12.1; SOURCE; UNZIP
Errors
________________________________________ORA-201
Thanks
Piyush Prakash
7 comments:
Hi there! This article could not be written much better!
Looking through this post reminds me of my previous roommate!
He continually kept preaching about this. I am going to forward this article to him.
Fairly certain he'll have a very good read. Many thanks for sharing!
My blog; real estate listings
First off I would like to say terrific blog!
I had a quick question which I'd like to ask if you do not mind. I was curious to know how you center yourself and clear your head before writing. I have had difficulty clearing my mind in getting my thoughts out. I truly do take pleasure in writing however it just seems like the first 10 to 15 minutes are generally wasted simply just trying to figure out how to begin. Any suggestions or hints? Appreciate it!
My web blog ... airbrush cartoon t shirts
Very good info. Lucky me I recently found your website by chance
(stumbleupon). I have saved it for later!
Stop by my web site :: research paper abstract example
Great post. I was checking constantly this blog and I'm impressed! Very helpful information particularly the last part :) I care for such information a lot. I was seeking this particular information for a very long time. Thank you and good luck.
my blog post: psychology research paper example
Thanks for sharing your thoughts about my essay. Regards
Feel free to surf to my web-site term paper presentation
Just wish to say your article is as astonishing.
The clarity in your post is simply spectacular and i can
assume you're an expert on this subject. Well with your permission let me to grab your RSS feed to keep updated with forthcoming post. Thanks a million and please continue the enjoyable work.
Feel free to surf to my site - help with essay papers
I don't even know how I ended up here, but I thought this post was good. I do not know who you are but certainly you are going to a famous blogger if you aren't already ;) Cheers!
Also visit my web site; USA real estate directories
Post a Comment