Hey FolksI'm configuring citrix Secure Private Access ( https://docs.citrix.com/en-us/citrix-secure-private-access.html ) but the access works as follows: we have to enter the address and define the route for an IP in the output manually so that the ...
I'm trying to ingest streaming HL7 messages and planned to use Kafka with Smolder to achieve this. I'm having a little trouble understanding how Kafka can run on databricks. So far I've installed Kafka and started the cluster but the start command ju...
@Jose Gonzalez​ Sorry for the delay. I've actually already looked at the provided links. The first one is for AWS. I'm currently using Azure. The KAFKA setup is working..I think? I'm stuck at ok so I have a server listening now what? How do I send/r...
Hi, Assume that I have created several tables in Databricks workspace. If two tables have similar names but stored in different folders, how I can read them by "spark.read.table". I ask this question because input to the "spark.read.table" is only ta...
spark.read.table reads table registered in catalog/metastore. If you have multiple databases in metastore, just prefix table name with database name db.table
AWS by the way, if that matters. We have an old production table that has been running in the background for a couple of years, always with auto-optimize and auto-compaction turned off. Since then, it has written many small files (like 10,000 an hour...
Hey there @Fenno Vermeij​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
With TACL enabled cluster, you got many restrictions, so streaming will not work. Generally, you can read only things registered in metastore; please disable it for your use case,Additionally, remember that the unity catalog doesn't support streaming...
Calling all Data Engineers! If you’re looking to join the most iconic brand in STL as they transform how business decisions are made then this Bud is for you!https://www.linkedin.com/posts/colt-hubbartt-90578b13_senior-data-engineer-activity-69783757...
I have found the Administrator Console. It is in the top right under my name. My ui is different.Thanks community for your support. I am a junior on this product.
Hi Everyone ,I am trying to run a databricks notebook in parallel using ThreadPoolExecutor .Can anyone suggest how to reduce the time taken based on the below findings so far.Current Performance:Time taken - 25 minutes ThreadPoolExecutor max_workers ...
ThreadPoolExecutor will not help as Databricks/Spark will process job by job.So please analyze in Spark UI what is consuming the most time.There are a lot of tips on how to optimize they depend on the dataset (size etc. transformations)Look for data ...
I am reading delta tables from my Databricks workspace as DF and then I am trying to write this DF into Azure Synapse Dedicated SQL Pool, but I am getting error like:-Py4JJavaError: An error occurred while calling o1509.save.
: com.databricks.spark.s...
The issue was getting created because I was passing SAS Token for setting the configuration for Storage Account instead of Storage Access Key. It's working now without any issue..spark.conf.set("fs.azure.account.key.datalakestorage.dfs.core.windows.n...
The Nova City Islamabad, is an upcoming and emerging housing project in the federal capital. This project aims to target all budget genres of people of Pakistan. In the future, this project is going to compete with the mega projects of Islamabad, Pak...
The New Metro City Gujar khan Payment Plan has not yet been made public, but the developers are anticipated to do so shortly. All of the plots will be within the budget and at reasonable pricing. The investment in this project will be simple and hass...