- 24119 Views
- 5 replies
- 7 kudos
Resolved! Best Practices for naming Tables and Databases in Databricks
We moved in Databricks since few months from now, and before that we were in SQL Server.So, all our tables and databases follow the "camel case" rule.Apparently, in Databricks the rule is "lower case with underscore".Where can we find an official doc...
- 24119 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @Salah KHALFALLAH​ , looking at the documentation it appears that Databricks' preferred naming convention is lowercase and underscores as you mentioned.The reason for this is most likely because Databricks uses Hive Metastore, which is case insens...
- 7 kudos
- 2405 Views
- 3 replies
- 3 kudos
Resolved! Why does chaining spark.read from one system/driver and .write to another system/driver take so much longer than doing each piece individually?
i am reading data from IBM DB2 and saving into a MS SQL server (the first step is moving the code itself to databricks, and then we will move the databases to databricks itself). Problem I'm running into is doing something like the below will take > ...
- 2405 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi, it is related to partitioning optimization. By default, the JDBC driver queries the source database with only a single thread. So write was from one partition as one partition was created, so it was using a single core. When you used pandas, it d...
- 3 kudos
- 2539 Views
- 4 replies
- 4 kudos
RStudio on Databricks user experience
Is anybody actually using the RStudio app integration on Databricks? I'm surprised to find so little discussion in this forum. My team has been using it for about 3 months and it seems under-developed.1) No automated backup, you have to do it yoursel...
- 2539 Views
- 4 replies
- 4 kudos
- 4 kudos
@Jonathan Dufault​ Thanks for the response, and glad I'm not alone. My problem (and this is probably just a preference thing) is that the 'reward' of using a full-fledged IDE is huge, compared to bouncing between notebooks in multiple tabs. The integ...
- 4 kudos
- 1865 Views
- 0 replies
- 0 kudos
Singleton Design Principle for pyspark database connector A singleton is a design pattern that ensures that a class has only one instance, and provide...
Singleton Design Principle for pyspark database connectorA singleton is a design pattern that ensures that a class has only one instance, and provides a global access point to that instance. Here is an example of how you could implement a singleton d...
- 1865 Views
- 0 replies
- 0 kudos
- 3891 Views
- 2 replies
- 2 kudos
How to use partial_parse.msgpack with workflow dbt task?
I'm looking for direction on how to get the dbt task in workflows to use the partial_parse.msgpack file to skip parsing files that haven't changed. I'm downloading my artifacts after each run and the partial_parse file is being saved back to adls.Wha...
- 3891 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, Could you please confirm what will be your expectation and the used case? Do you want the file to be saved somewhere else?
- 2 kudos
- 4213 Views
- 4 replies
- 7 kudos
Resolved! Connecting azure synapse through data bricks note books
Hi All, Happy new year!how can we connect to azure synapse serverless sql pool through databricks notebooks and execute DDLs
- 4213 Views
- 4 replies
- 7 kudos
- 7 kudos
@KVNARK .​ https://joeho.xyz/blog-posts/how-to-connect-to-azure-synapse-in-azure-databricks/
- 7 kudos
- 3744 Views
- 2 replies
- 2 kudos
Read/Write concurrency issue
Hi. I assume that it can be concurrency issue. (a Read thread from Databricks and a Write thread from another system)From the start:I read 12-16 csv files (approximately 250Mb each of them) to dataframe. df = spark.read.option("header", "False").opti...
- 3744 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Anastasiia Polianska​,I agree, it looks like a concurrency issue. Very possibly this concurrency problem will be caused by an erroneous ETAG in the HTTP call to the Azure Storage API (https://azure.microsoft.com/de-de/blog/managing-concurrency-in...
- 2 kudos
- 5462 Views
- 2 replies
- 4 kudos
- 5462 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @maddy v​ ,I recommend that you use the Databricks SQL module for this type of reports and email alerts. It is a very interesting module with multiple options for your use case.https://learn.microsoft.com/en-us/azure/databricks/sql/user/dashboards...
- 4 kudos
- 1290 Views
- 1 replies
- 3 kudos
Spark SQL & Spark ML
I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank you ...
- 1290 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi, please refer https://docs.databricks.com/machine-learning/index.html, please let us know if this helps.
- 3 kudos
- 1314 Views
- 1 replies
- 2 kudos
I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark...
I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank y...
- 1314 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, please refer https://docs.databricks.com/machine-learning/index.html, please let us know if this helps.
- 2 kudos
- 1559 Views
- 1 replies
- 2 kudos
How to Enable Files in Repos in DBR 7.3 LTS ML ?
we need a custom version of a GPU cluster with following requirements for a certain project:Ubuntu 18.04Cuda 10.1.Tesla T4 GPU.Availability of /Workspace/Repos folder.All of these requirements are available with DBR ML 7.3 LTS. But one critical compo...
- 1559 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, To work with non-notebook files in Databricks Repos, you must be running Databricks Runtime 8.4 or above.https://docs.databricks.com/files/workspace.html#configure-support-for-workspace-files
- 2 kudos
- 1337 Views
- 1 replies
- 2 kudos
How to access image data for yolov5 object detection module
How to access image data for the yolov5 object detection module, I am unable to do so.
- 1337 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, Could you please provide your used case briefly on the same?
- 2 kudos
- 2755 Views
- 1 replies
- 2 kudos
How to find the road distance between two cities? We can use Azure databricks and azure map.
We will be given an excel file, in which we can see column sender_city and destination_city. We have to find the distance between these two cities and the distance calculated should be written in a column total_distance. All these processes should be...
- 2755 Views
- 1 replies
- 2 kudos
- 2 kudos
heywithout using latitude and longitude it is hard to find out but you can try some distance-based algorithm
- 2 kudos
- 3155 Views
- 2 replies
- 3 kudos
Auto Loader: Empty fields (discovery_time, commit_time, archive_time) in cloud_files_state
Hi! Why are the fields discovery_time, commit_time, and archive_time NULL in cloud_files_state? Do I need to configure anything when creating my Auto Loader? df = spark.readStream.format("cloudFiles") \ .option("cloudFiles.format", "json") \ ...
- 3155 Views
- 2 replies
- 3 kudos
- 3 kudos
Please be sure that the DBR version is 10.5 or highercommit_time and archive_time can be null but discovery_time is set even as NOT NULL in the table definition so it is a bit strange. Please change the DBR version first.
- 3 kudos
- 3465 Views
- 3 replies
- 4 kudos
Resolved! Bug in Delta Live Tables when missing files option?
When using Delta Live Tables with SQL-syntax ignoreMissingFiles-option is not working and you are getting error anyway.(See picture below)Link to feature: https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/options#generic-option...
- 3465 Views
- 3 replies
- 4 kudos
- 4 kudos
You could also use inferSchema. ignoreMissingFiles option is to handle files that were accidentally deleted before being fully processed, so it has nothing related to the schema.
- 4 kudos
| User | Count |
|---|---|
| 1615 | |
| 777 | |
| 349 | |
| 287 | |
| 260 |
