AX for Retail 2012 R2: Troubleshooting the Real-time Service

AX for Retail 2012 R2: Troubleshooting the Real-time Service

This is a follow-up from my previous post AX for Retail 2012 R2: Installing the Real-time Service and should give you some tips if you run into problems installing and configuring the Real-time Service.  Before getting into those tips, I want to go through the configuration of logging, as that will be necessary for your troubleshooting.

Edit (2013-03-29):   Added a couple more common errors and a little more detailed information about self-signed certificates.

Edit (2013-06-21):  A minor revision on configuring logging

Configuring Real-time Service Tracing and Logging

The Real-time Service has robust tracing and logging that is easy to configure.  Even if you have no problems getting the service up and running, there will be times where you need to dig into POS errors.  Remember that the Real-time Service is a conduit between the POS and the X++ logic that is running in AX.  When errors happen in the X++ code, the POS will often display and log a rather generic error.  Getting to the root of the issue usually requires digging into the X++ errors.  These are logged by the Real-time Service itself.

Configuration of the Real-time Service logging happens in the web.config file for the Web service.  If you installed the service with the Powershell script, this will be in thec:\inetpub\dynamicsaxretail\CommerceDataExchangeRealtimeService\ folder.  Load the web.config file into any text editor and note these two lines (they should be lines 14 and 28 by default):

<source name="RetailNetTracer" switchValue="Warning">

<add name="RollingXmlWriterTraceListener" type="Microsoft.Dynamics.Retail.Diagnostics.RollingXmlWriterTraceListener, Microsoft.Dynamics.Retail.Diagnostics" initializeData="" MaxLogFileInBytes="50000000" traceOutputOptions="ProcessId, DateTime, LogicalOperationStack" />

The easiest way to trace what is happening in the Real-time Service is by using the RollingXmlWriterTraceListener.  This is a common log format that is also used by other components of AX for Retail. 

Line 14:  The value in the switchValue property is the level of tracing.  Common levels that I like to use include “All”, “Warning”, and “Error”.  Note:  these values are case-sensitive.  If you spell it incorrectly, your Real-time Service will immediately break when you save the web.config file!

Line 28:  The value of the initializeData is where you can specify a folder to which the log file will be written.  If this property is left blank it can be difficult to find the file as it will be stored in the<windows>\temp\RetailLogs folder.  I like to save the file somewhere easier to find like c:\logs.

Note (2013-06-21):  It is better to use a path + filename here:  "c:\logs\RTS.svclog" for example.  This will result in a filename of "c:\logs\RTS_00.svclog" being created.  Note that you can use the .svclog extension so you can just double-click on the file to open it in the Service Trace Viewer.

Also note that the folder you specify has to already exist and the Application Pool user account must have write privileges to the folder.

Although the files are standard XML files, they are messy to read in an XML viewer.  Instead, you should use a small program called “MICROSOFT Service Trace Viewer” (SvcTraceViewer.exe) which is a standard tool for reading these types of log files.  That program expects extensions of “.svclog” but you can do a File > Open to open the XML files.  Once loaded, click on the “Activity” section on the left and then you will see all of the messages on the right, very similar to how they would appear in Event Viewer:

Depending on your trace level, these messages will also include exceptions and warnings from X++ methods that the service is calling.

Here is a good blog post on installing SvcTraceViewer.  The application itself can be a bit tricky to get ahold of since you need to pull it from the Windows SDK, but once you find it you can just copy the .exe to any machine and run it directly.

Common Install and Configuration Issues

Problem:  The InstallCommerceDataExchangeRealtimeService Powershell script fails with an error “PathNotFound” on the web.config file:

Solution:  The most common reason for this failure is by not specifying a valid ServiceBinarySourceFolder parameter when running the script.  Most often, a trailing “\bin” is added.  The correct path should be:  “c:\Program Files (x86)\Microsoft Dynamics AX\60\Commerce Data Exchange\Real-time Services\6.2\”.  You should run the uninstall script and then re-run the install script with the correct parameter.

Problem:  When testing the WCF application in the Web browser you see this error:  Invalid hexadecimal string format

Solution:  Verify that you entered the Thumbprint for your server certificate correctly.  It should be 40 characters with no spaces.  It is also very susceptible to copy and paste issues (hidden characters, ascii/unicode character sets, etc.) so it may be helpful to type it in manually.  The thumbprint is stored in the web.config file in the <serviceCredentials>node, but it may be best to run the uninstall and install using the Powershell script if you need to change it.

Problem:  When testing the WCF application in the Web browser you see this error:  Could not find a base address that matches scheme net.tcp for the endpoint with binding NetTcpBinding.  Registered base address schemes are [http,https].

 Solution:  We have seen a couple instances where the net.tcp binding may not get created by the Powershell installation script.  In the IIS Manager, go to your CommerceDataExchangeRealtimServiceWebsite note and right-click and select bindings: 

Verify that net.tcp is included with binding information of 808:*.

Under the application itself (CommerceDataExchangeRealtimeService), right-click and select Manage Application > Advanced Settings.  Verify that net.tcp is included:

Problem:  Call to the Real-time Service results in an error similar to (POS error log): System.IdentityModel.Tokens.SecurityTokenValidationException: The X.509 certificate CN=TSServerCert chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. A certificate chain could not be built to a trusted root authority.

Solution:  One of the more difficult tasks in configuring the Real-time Service is getting your Server Certificate set up correctly.  If you are creating a test environment it is very likely that you are using a test or self-signed certificate.  If this is the case, all clients attempting to connect to the Real-time Service (i.e. your POS machines) need to create a trust chain with that server.  This step is not needed if you are using a purchased certificate because the publisher of that certificate is already a trusted authority.

If you find yourself in this situation, you can run the InstallCertificationAuthority Powershell script that is included with the Real-time Service installation.  You will need a .cer file (certification authority) and a .crl (revocation list) to run the script.

Alternatively, you can install the certificate on the POS machine itself: 

1. Export the certificate from the Certificates > Computer account (see http://technet.microsoft.com/en-us/library/cc779668(v=WS.10).aspx for details)
2. Copy the resulting .cer and/or .pfx files to the POS machine.
3. Double-click each of the files and follow the Certificate Import Wizard to install the certificate.

Note:  Keep in mind that you should not use a self-signed certificate in a production environment.  When you purchase your certificate from a trusted authority the POS machines will automatically create a trusted connection to the Real-time Service without having to install certificates.

  

Problem:  Call to the Real-time Service results in one of these errors (POS error log):  TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond <ip>:808 or TCP error code 10061: No connection could be made because the target machine actively refused it 10.11.12.13:808

Solution:  In the Services console on the Real-time Server machine, verify that Net.Tcp Listener Adapter and Net.Tcp Port Sharing Service services are running.

Problem:  Call to the Real-time Service results in an error similar to: Exception: The requested service, 'net.tcp://localhost/RetailTransactionService/Service.svc/Common' could not be activated. See the server's diagnostic trace logs for more information. [CryptographicException: Keyset does not exist]

Solution:  This can occur if the user account that the Real-time Service is running under does not have access to the server certificate private key.  In IIS management console, first identify the identity for theCommerceDataExchangeRealtimeServiceAppPool application pool.  Open the Certificate Manager console (start –> type in mmc, add the snap-in for certificate manager, local machine).  Expand Personal –> Certificates node.  Right click on the certificate, choose All tasks -> Manage private key...  Add read access for the user identified above.

Problem:  Call to the Real-time Service results in an error similar to:  Could not load type 'System.ServiceModel.Activation.HttpModule' from assembly 'System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.

Solution:  This can happen if the .Net Framework 4.0 gets unregistered from IIS for some reason. This registration should occur with the installation of the Real-time Service, but you can run the following command in an elevated command prompt to manually register:  %WINDIR%\Microsoft.NET\Framework64\v4.0.30319\aspnet_regiis.exe -i

Problem:  Call to the Real-time Service results in an error similar to the following (POS error log): RetailPosis.TransactionServices.EstablishConnection: System.ServiceModel.Security.MessageSecurityException: Identity check failed for outgoing message. The expected DNS identity of the remote endpoint was 'TSServerCertX' but the remote endpoint provided DNS claim 'TSServerCert'. If this is a legitimate remote endpoint, you can fix the problem by explicitly specifying DNS identity 'TSServerCert' as the Identity property of EndpointAddress when creating channel proxy.

Solution: In HQ, verify that the Common Name on the Real-time Service profile your POS is using matches the Common Name of your server certificate:

Problem:  Call to the Real-time Service results in an error similar to the following (POS error log):  Inventory lookup: LSRetailPosis.PosisException ---> System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. ---> System.ServiceModel.FaultException: An error occurred when processing the security tokens in the message. In addition, an error similar to the following is found in the Real-time Service log:  Username password validation failed: The password is null or whitespace

Solution:  In HQ, verify that you have entered a value in the Passphrase field of the Real-time Service profile.  Note that the field may look populated with the mask character (dots) even if you have not entered one.  You do not have to enter this passphrase anywhere else as it gets pushed to the POS database via the N-1090 job.  See Brett’s article for more information about how the Passphrase field works:  Unable to connect to the Transaction Service - The case of the missing or mismatched passphrase.

Problem:  Call to the Real-time Service results in an error similar to the following (POS error log): LSRetailPosis.TransactionServices.EstablishConnection: System.ServiceModel.ProtocolException: You have tried to create a channel to a service that does not support .Net Framing. It is possible that you are encountering an HTTP endpoint. ---> System.IO.InvalidDataException: Expected record type 'PreambleAck', found '72'.

Solution:  This usually happens when you are mixing protocols and ports incorrectly in the Real-time Service profile.  For instance, I can force this error if I choose "net.tcp" but use port 9080 (which is the http port).  We recommend that you use "net.tcp" with port 808 for the communication between POS and the Real-time Service.

Problem:  Call to the Real-time Service results in an error similar to the following (POS error log): LSRetailPosis.TransactionServices.EstablishConnection: System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.tcp://localhost/xCommerceDataExchangeRealtimeService/Service.svc/Common that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

Solution:  This is almost always caused by a mismatch between the "Web application name" setting on the Real-time Service profile and the actual application itself.  In my example I intentionally misspelled it; usually when this error pops up I see people using the Web site itself (CommerceDataExchangeRealtimeServiceWebsite) instead of the application (CommerceDataExchangeRealtimeService).  Double check those two and make sure they're exactly the same.

Problem:  The POS cannot connect to the Real-time Service and the Real-time Service log has an error similar to the following:  Axapta.Logon() failed. Exception details: The server is unavailable. Microsoft Dynamics will try to connect to other servers in your configuration.Microsoft.Dynamics.BusinessConnectorNet.ServerUnavailableException:

Solution:  There are a number of possibilities here.  First, if you have overridden the AOS connection in the Real-time Service’s web.config, make sure that it is pointing to a valid AOS.  Also, make sure that this AOS is up and running.  Finally, verify that the Real-time Service user account (In IIS management console, identify the identity for theCommerceDataExchangeRealtimeServiceAppPool application pool) is a valid AX user that has rights to connect via the .Net Business Connector.

Hopefully these tips will get you up and running with the Real-time Service and able to track down issues when the POS errors don’t