U2 Dictionaries [Part 1]

February 5, 2010 Leave a comment

Something that often gets overlooked in the U2 world is best practice regarding dictionaries.

Before I get into it however, a very brief introduction to dictionaries for those who are new to UniVerse and UniData.

SQL databases have a schema which defines what data can be stored and where it is stored in relation to the rest of the data. This means every bit of data has a related section in the schema with gives it a name and a type.

UniVerse and UniData do not do this. The schema (dictionary) is simply there to describe the data (as opposed to define). You can give sections of the data arbitrary names and/or data types. In fact, you can give the same location multiple names and types, or even create a dictionary item that describes multiple other sections! Each file uses another file called a dictionary to hold its ‘schema’ (Which, for the rest of this post, will no longer be called a schema since it is misleading).

According to the UniData “Using UniData” manual, it describes a dictionary as containing “a set of records that define the structure of the records in the data file, called D-type records”. Now, it is very important to remember this next point: The manual is at best overly optimistic and at worst flat-out lying.

In SQL (excluding implementations such as SQLITE), if you get a table schema and it informs you that the third column is an INTEGER and it was called ‘Age’, then it would be safe to assume it was what it said it was. In the worst case, you can be certain it won’t ever return a value of “Fred”. In UniVerse and UniData, the dictionary doesn’t even need to contain a record to describe the third attribute (an attribute is sort of like a column, but different).

Also of note to new players is that D-type records are not the only records in a dictionary file. There are 3 other types of records to consider. Once again, straight from the manual: ‘A dictionary may also contain phrases, called PH-type records, and items that calculate or manipulate data, called virtual fields, or V-type records. A user may also define a dictionary item to store user-defined data, called X-type records’.

What does this mean for you? Well, like most of U2, when looking at the records in a dictionary file, anything goes. Some could be accurately describing the file structure, while others could be getting fields from sections in a completely different file. Some again could have nothing to do with the data in the file at all and are merely there because a programmer has used it as a convenient dumping ground. Also to consider is the item being completely wrong.

There are 2 sides to this. 1) It can make development faster as you can just tack on extra bits of data with no maintenance work required. 2) As a result of 1, you can quickly find systems in a state where your records have mystery data and you cannot even begin to work it out without scouring through many programs and manually inspecting the data.

Even more confusing, is that you can have multiple records referring to the exact same location but describing the data differently.

If you were to describe the U2 system holistically, you could call it a weakly-typed system. Whereas some other databases, query languages and programming languages are strongly-typed.

This is where best practices come in. Here are several simple rules, that if followed, should go a long way to ensure your dictionary files are useful, accurate and easier to maintain.

  1. If you ever add new data to a record, then you MUST create at a minimum, a D-type record to describe each piece of data.
  2. Always check that an appropriate dictionary item doesn’t already exist before creating a new one to reference a section of data in an existing file
  3. If you come across a missing dictionary item, don’t ignore. Either create it or add it to whatever bug-tracking system you use.
  4. Remember, after the type in attribute 1, you can write anything. Use this to describe what the data is if the name isn’t sufficiently self descriptive.
  5. Also, if the data is effectively a foreign key for another file, use the end of attribute 1 to mark it as such (including the main file it references).
  6. Use the User-type record ability to add a single record that describes the general purpose/usage of the overall file. Give it a simple/recognisable name like README or FILEINFO

Improving the Community

January 29, 2010 2 comments

I’ll admit it. I was quite sceptical when I was first introduced to UniData. It was different from other systems I was familiar with. It also seemed quite antiquated. It didn’t help that I was introduced to it via a green screen and single line editor that was preached as “very powerful” because it could do the equivalent of ‘Find & Replace’

It does have its charm, however. It also exceeds in areas where other solutions don’t.

The biggest problem I still have with U2 over other stacks out there, is the community. Now, the problem isn’t the people in the community. The problem is size and accessibility of the community compared to the mainstream stacks. This is especially true for newer users/developers as their normal sources of information generally pull up empty regarding U2.

StackOverflow (programming), ServerFault (Server admin) and SuperUser (End-User) have quickly become one of those “One-Stop-Shops” for Q&A’s for large number of admins, developers and users. I am one of those developers. I even link to it on the right-side here. I have not gone there for U2 answers though, since I know there is only a small handful who know about UniData or UniVerse that frequent it.

Evan Carrol recently posted about StackOverflow/ServerFault as an alternative medium. I completely agree and find that while the mail-list serves it’s purpose, I found the above trilogy of sites to have a far more useful system in place. I also believe that more activity at these sites will be far more useful and easy to find for new players.

From now one, if I post a question on the mailing list, I will also post on the appropriate above site. I also encourage any others in the U2 world to do the same.

That isn’t the only problem facing the community when competing with the mainstream stacks. U2 has a unique business model which causes users to not only be isolated from the actual U2 team by “middlemen” (resellers, etc) who act as first line support, but also isolating the users from each other. The U2 User Group helps, but I feel still doesn’t negate this initial segregation.

Comprehensive introduction tutorials (not just for the core product, but also the additional bolt-on products), a more direct route for reporting bugs to Rocket and an easier to use community site would go along way to closing the gap.

That’s my opinion anyway.

Categories: Community Tags: ,

Security is not Obscurity. Even in U2 [Part 2]

January 22, 2010 Leave a comment

I hope everyone had a relaxing and enjoyable holiday season!

I also hope everyone is considering what information they log. Is it too little for when things go wrong? Is it too much and compromising your security?

Lets start with too little.

Say you have a system that services requests from an external system. Maybe it does login requests as well. Now, you need to explain something that has changed in your data. Do you have log files with enough information to look back on and determine when the change happened and who made the change? Do you have logs at all?

Without appropriate logs, you leave yourself vulnerable to unexplainable and unaccountable changes. Effective logging not only gives you diagnostic capabilities for when mistakes happen (user OR developer), but can also act as a deterrent for would-be malicious parties.

On the flip side, it is possible for too much information to be logged.

Aside from the administration headache causes by excessive amounts of data, it is possible that you are recording normally secure information in unsecured log files.

Typically, log files have security practices that are much more lax than the most secure data on your system. Log files are more likely to be found on developers machines, in print-outs and generally unencrypted.

With that in mind, you must make sure that sensitive information is effectively censored from logs. This means information such as passwords, credit card details and clients personal information. This should be done in the program that creates the data to be logged, as it should know which sections should be censored. If this isn’t possible, the log files should be censored by another program before being accessible by anyone else.

Think of any programs you have that do custom logging or create printer spools. Don’t forgot to check log files that you can get UD/UV to automatically create for you such as COMO logs, Protocol logging and system logs. Also important to consider is the logging done by applications that talk to your U2 system.


When considering security of Data in Motion and Data at Rest, it is highly likely that it is at rest in more places than just your database. Sure, may have your U2 system locked up tight, but if the information unscrupulous parties are after can be accessed elsewhere, you can bet they will just get it from the easier target.

Categories: Security Tags: ,

Security is not Obscurity. Even in U2 [Part 1]

December 6, 2009 2 comments

Sure, we may benefit from whatever shelter is derived from running a less widely used/understood system, but relying solely on security through obscurity with your U2 system is, to put it nicely, extremely naive in this day and age. It just doesn’t add up when you pay thousands of dollars for firewalls and other network security paraphernalia and wouldn’t dream of allowing raw user input through in your SQL-based applications.

U2 may have different syntactical spices and a different method of representing data than its mainstream counterparts but the core principles behind secure coding practices still apply:

The list goes on.

So, what specific vulnerabilities should we look out for?

Let us start with the humble EXECUTE/PERFORM statements in UniData and UniVerse. SQL Injection is a widely know subject, but how many U2 developers have considered UniQuery/RetrieVe Injection? Did you know that in some cases, malformed UniQuery in a EXECUTE can drop you to ECL?

As developers in the U2 world, the same lessons learnt in SQL Injection can and should be applied when using *EXECUTE/*PERFORM, etc. Sanitise your input!

Do you have any statements that work like this?


In this case, TAINTED.INPUT is either supplied by a user or is comes from an external source. The results from the SELECT statement are now compromised and can contain any data. Take for instance the following input for this contrived example:

" OR WITH CC.NUMBER = "4657000000000000

Essentially, this converts the innocent SELECT statement which, for example, was used to search for customer’s first names to get contact numbers, into one which can be used to find Credit Card numbers (hopefully though, your CC numbers are encrypted in some manner). Even worse, if your program displays error messages that reveal record names when they cannot be read, then an attacker with patience can reveal almost any data they want from your system.

Remember, in UniData you can use the ‘USING’ keyword to specify any FILE to source the dictionaries (UniVerse does not have this, I believe). Aside the all the usual manipulation of results, the USING means that if someone can control the first few lines of a record (temp data dumping file anyone?) then using the SUBR() call, they can even cause programs and subroutines to be called!

Before you EVER use input from a user or an external source, make sure it is validated and sanitised. Expecting a number? Use MATCH and 1N0N. Expecting a name? Make sure it is doesn’t contain double quotes. Don’t want to allow ‘searching’ with your SELECT? For example, I use the following check to ensure the user input doesn’t escape the SELECT string with double quotes or attempt a wildcard search with [ or ].


Further to this UniQuery/RetrieVe Injection vulnerability, earlier I mentioned that in certain situations you could cause it to crash to ECL.

Are you running UniData in PICK mode? If you are, I suggest you type ‘UDT.OPTIONS’  at ECL right now and continue down until you see whether ’41  U_UDT_SERVER’ is set to ON or OFF. Now, did it say OFF? If so, then read on because you may be vulnerable.

While that option is turned off, certain malformed UniQuery statements can cause you to crash straight to ECL, even if you are in a program called by a program.

Lets see an example. First, compile the following program in Pick Mode.

CRT "Enter the program to select"
CRT "Executing query..."
CRT "Checking results..."
   CRT "Program FOUND!"
   CRT "Program doesn't exist"

Now, when you run the program (I called it CRASHTEST), put a record ID that exists in BP. Try again with a record ID that doesn’t exist. Your results should be something like this:

Normal Program Operation

Normal Program Operation

Looks good. Program works as expected. Underneath this simple program though, lies a bug feature in the Pick Parser. To show this, I will use a specifically formed input that will make the programs SELECT malformed in a manner to gain ECL access. This time when you run the program (with UDT.OPTIONS 41 off) type in this input, including quotes:
" @ID="
In this case, the program will crash out before ever returning.

Crashed from EXECUTE

Crashed from EXECUTE

There are 2 ways to deal with this. The first way is to set UDT.OPTIONS 41 to ON. This will result in the EXECUTE returning so we can handle it whatever way we wish.

The other way is to set ON.ABORT. I created a VOC paragraph for this, called EXCEPTION as follows:

DISPLAY This program aborted

Running the same test above now results in:

Aborted program

Aborted program

Personally, I believe the method that returns control to the program (UDT.OPTIONS 41) and handles it accordingly (always check what was set by RETURNING) is the safest since it doesn’t give away that the program has a compromised EXECUTE statement. However, this may not always be a viable option, so make sure you at least have ON.ABORT set.

Xalan errors in U2 XML

December 1, 2009 Leave a comment

Just a quick note about errors from U2 XML. I’m assuming UniVerse uses the same XML parser as UniData.

I was playing around with converting XML to HTML using XDOMTransform yesterday when an error got returned. After using XMLGetError I could see it was an ‘uncaught XalanDOMException’ (code 14 to be exact, UD error code 16.) That meant exactly zilch to me, so I looked up the Xalan documentation and found the following:

XalanDOMException Class Reference

Turns out it was unhappy about my namespace attribute in the html tag inside my xslt file.

Maybe this reference will be useful to someone else if you encounter other XalanDOMException errors from U2.

Statement Code Coverage Testing

November 27, 2009 1 comment

On the right-hand side you will see a link to the “Unibasic Code Coverage” project on sourceforge. This tool will enable you to perform statement level code coverage tests on UniData code. The results are then tabulated and saved as a colour coded HTML file.

This is based on the original prototype I wrote before further developing it for my current employer. Although this open source version is relatively simplistic, I will be progressing it by back-porting features as well as trialling new ideas.

If you try it out, leave a comment or send me an email and let me know how it went. If you are interested in helping with the development of it, or making the minor changes needed to port it to UniVerse then get in touch!

Introducing… U2Tech.

November 26, 2009 Leave a comment

U2Tech is the place where you can see my thoughts on Development in the U2 world. From what I love about it to what really irks me. Although I only professional work with UniData, the sister system UniVerse won’t be left in the cold.

Not only will there be my potentially inane ramblings, but also posts on development progress of Open Source Tools for U2 and some fun coding challenges. I’ll be talking about secure development for U2 and will touch on performance considerations for those time/space critical sections of your system.

It is good to see the U2 team furthering the tools available to developers, but U2 still has a long way to go in catching up to other modern languages.

Hopefully we can help.

Categories: Community Tags:
%d bloggers like this: