- 2419 Views
- 3 replies
- 0 kudos
Ingestion Time Clustering on initial load
We are migrating our data into Databricks and I was looking at the recommendations for partitioning here: https://docs.databricks.com/tables/partitions.html. This recommends not specifying partitioning and allowing "Ingestion Time Partitioning" (ITP)...
- 2419 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @dvmentalmadess Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you. T...
- 0 kudos
- 1321 Views
- 0 replies
- 0 kudos
Connect to EXCEL
> I have hard time connecting to Excel, any help connecting Data Bricks to EXCEL
- 1321 Views
- 0 replies
- 0 kudos
- 2634 Views
- 1 replies
- 0 kudos
Resolved! Databricks SQL Statement Execution API
I'm trying to follow the example provided here https://github.com/databricks-demos/dbsql-rest-api/blob/main/python/external_links.pyIt fails when it comes to downloading the data chunks. The statement status turns from SUCCEEDED to CLOSED right away ...
- 2634 Views
- 1 replies
- 0 kudos
- 0 kudos
It turns out that if the response is small and can fit 16mb limit, then status check will also provide single external link to download the data.So I need a condition here. Maybe even something like thisif len(chunks) == 1: external_url = respons...
- 0 kudos
- 2476 Views
- 2 replies
- 2 kudos
How to install cmdstanpy in dbx cluster
I have built an HMC model using cmdstand. In my local machine, I have install cmdstan for the following approach. import cmdstanpy cmdstanpy.install_cmdstan()But in Databricks I need to reinstall it every time when I train a new model, from the noteb...
- 2476 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Rajamannar Aanjaram Krishnamoorthy​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 1907 Views
- 1 replies
- 1 kudos
Creating a test schema - what is the best practice?
Hey, I've created a schema with few tables with historical data (prod), now I would like to have a Dev/testing environment with exactly the same data.What do you recommend? CTAS? Shallow clone? Deep clone? I wonder if shallow clone would be sufficien...
- 1907 Views
- 1 replies
- 1 kudos
- 1 kudos
@Asterol If you would like to have the same data for your Dev/testing environment, I would recommend using Deep Clone. Deep clone copies the metadata and creates an independent copy of the table data. Shallow clone only copies the metadata and will h...
- 1 kudos
- 2136 Views
- 0 replies
- 0 kudos
Sample code to read json from service bus queue in Azure
Hi,I'm looking for sample notebook or code snippet to read messages from Azure Service bus queues. I looked for documentation couldn't find anything. Any help would be appreciated. First we are thinking of batch mode before we move on to Streaming. P...
- 2136 Views
- 0 replies
- 0 kudos
- 2988 Views
- 4 replies
- 2 kudos
Resolved! Community Edition Sign-up Issue
Hi, I'm trying to signup community edition for learning purpose. The sign up page has issue in selecting country. The select dropdown doesn't work and continue option redirects to same page. Couldn't complete signup. Kindly help!
- 2988 Views
- 4 replies
- 2 kudos
- 2 kudos
for me it is alwas displaying below error but entered correct user name and passowrd,my user name : amit.das2k16@gmail.com Invalid email address or passwordNote: Emails/usernames are case-sensitive
- 2 kudos
- 16918 Views
- 11 replies
- 4 kudos
Resolved! Mongo Spark Connector 3.0.1 seems not working with Databricks-Connect, but works fine in Databricks Cloud
On latest DB-Connect==9.1.3 and dbr == 9.1, retrieving data from mongo using Maven coordinate of Mongo Spark Connector: org.mongodb.spark:mongo-spark-connector_2.12:3.0.1 - https://docs.mongodb.com/spark-connector/current/ - working fine previously t...
- 16918 Views
- 11 replies
- 4 kudos
- 4 kudos
Hi everyone the solution for me it was to replace spark.read.format("mongo") by spark.read.format("mongodb") my spark version is 3.3.2 and my mongodb version is 6.0.6 .
- 4 kudos
- 2565 Views
- 4 replies
- 1 kudos
Deploying existing queries and alerts to other workspaces
I have several queries and associated alerts in a workspace, and I would like to be able to deploy them to an other workspace, for example an higher environment. Since both queries and objects are not supported in repos, what is the way to go to easi...
- 2565 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @erigaud We haven't heard from you since the last response from @btafur , and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to others. Also, ...
- 1 kudos
- 943 Views
- 0 replies
- 0 kudos
JDBC DatabaseMetaData.getCatalogs()
Calling the DatabaseMetaData.getCatalogs() returns 'spark_catalogs instead' of 'hive_metastore', when connected to tradition version of databricks cluster which is not signed with uc_catalog tag.Please check this.
- 943 Views
- 0 replies
- 0 kudos
- 1750 Views
- 1 replies
- 0 kudos
Data Engineering Professional Certificate
On the way to obtain the certificate. Any preparing tips would be appreciated! Thanks!
- 1750 Views
- 1 replies
- 0 kudos
- 0 kudos
Disclaimer - I haven't taken this exam yet A couple of suggestions (from this forum, google searches, etc):- check out this blog post - https://medium.com/@sjrusso/passing-the-databricks-professional-data-engineer-exam-115cccc90aba#:~:text=I%20recent...
- 0 kudos
- 18171 Views
- 3 replies
- 2 kudos
Resolved! Union Multiple dataframes in loop, with different schema
With in a loop I have few dataframes created. I can union them with out an issue if they have same schema using (df_unioned = reduce(DataFrame.unionAll, df_list). Now my problem is how to union them if one of the dataframe in df_list has different nu...
- 18171 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi,I have come across same scenario, using reduce() and unionByname we can implement the solution as below:val lstDF: List[Datframe] = List(df1,df2,df3,df4,df5)val combinedDF = lstDF.reduce((df1, df2) => df1.unionByName(df2, allowMissingColumns = tru...
- 2 kudos
- 7144 Views
- 0 replies
- 0 kudos
Is there any way to convert delta share short-lived presigned URLs to CSV files at Client End
Hello All, I have requirement , where I need to disclose the data at the client end and they are suppose to access the data in CSV format. I am planning to use Delta Sharing integrated with Unity Catalog. As we know, according to Delta sharing protoc...
- 7144 Views
- 0 replies
- 0 kudos
- 2609 Views
- 3 replies
- 1 kudos
Data Engineering Professional - Practice exam?
Hi, when will we get Practice Exams for this the Data Engineering Professional Certification Exam? It seems like we already have it for a good amount of the associate exams, and this Professional exam seems more difficult than the associate ones, so ...
- 2609 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @180122 Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!
- 1 kudos
- 4335 Views
- 3 replies
- 3 kudos
Resolved! Questions about Lineage and DLT
Hey there!1. Does column lineage work across multiple catalogs and schemas?2. Do Delta Live Tables support lineage? If yes does that work across multiple pipelines or only with a single one?
- 4335 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @bradleyjamrozik We haven't heard from you since the last response from @Vinay_M_R and @erigaud , and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be he...
- 3 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
