- 430 Views
- 0 replies
- 0 kudos
The summit was great #Summit22​Â
The summit was great #Summit22​
- 430 Views
- 0 replies
- 0 kudos
Forum Posts
- 1659 Views
- 1 replies
- 0 kudos
Is there a way to watch a forum post ?
Is there a way to watch a forum post ? I cant see one.
- 1659 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Thanks, I was just wondering if there was a watch with notification function.
- 0 kudos
- 2616 Views
- 4 replies
- 0 kudos
Resolved! Add custom tags to DLT cluster
We have a policy in our AWS account that whenever an EC2 instance is created, there are 5 mandatory tags that need to be added. When we create an All-Purpose cluster in the Databricks console, we can easily add these under Advance Options > Tags. How...
- 2616 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
@Tomasz Bacewicz​ Perfect!! worked like a charm...thank you!!do you think auto loader or DLT is a better approach to implementing a CDC pipeline in Databricks?
- 0 kudos
- 5564 Views
- 3 replies
- 3 kudos
Delta Live Table Pipeline Load Error
I am getting this error when running Delta Live Table Pipelines pulling from a source in our sandbox's local dbfs folder. It says the user is not authorized to perform this operation, whereas I am able to see the data when I run a simple select state...
- 5564 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
Hey there @Abhay Sudhakaran​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution and mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...
- 3 kudos
- 2809 Views
- 3 replies
- 8 kudos
Resolved! AWS Private Link for DB workspace
What's the best way to set up a private link connection to Databricks without loosing the current settins? workspace
- 2809 Views
- 3 replies
- 8 kudos
Latest Reply
- 8 kudos
At the data and AI summit, there was a nice talk about AWS private link. Here is a link if you have access:https://dataaisummit.com/session-virtual/?v2477da705118cc74fd14460db021e1784e2eed5a7982c6482ec95cb2e86d259644b8741959f52a49e0e6908b82a9d860=C22...
- 8 kudos
- 3353 Views
- 3 replies
- 3 kudos
Resolved! How do I setup the right cluster?
- 3353 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
For general cluster decision making refer to this article https://docs.microsoft.com/en-gb/azure/databricks/clusters/cluster-config-best-practicesOnce you've selected a cluster that makes sense, run it and check your ganglia metrics to see whether yo...
- 3 kudos
- 3721 Views
- 5 replies
- 2 kudos
Resolved! DLT Pipeline and Job Cluster
We have written few python functions(methods within a class) and packaged them as a wheel library.In the as-is situation we use to install that wheel library in All-Purpose cluster that we already have created. It works fine.In the to-be situtation(D...
- 3721 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
Does it give you an error when running the DLT pipeline specifically on the %pip command or does it not work in some other way? If it's the former, could you share the path format that you're using for the %pip command path?
- 2 kudos
- 5785 Views
- 3 replies
- 4 kudos
Resolved! Spark persistent view on a partition parquet file
In Spark, is it possible to create a persistent view on a partitioned parquet file in Azure BLOB? The view must be available when the cluster restarted, without having to re-create that view, hence it cannot be a temp view.I can create a temp view, b...
- 5785 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
Here is what worked for me. Hope this helps someone else: https://stackoverflow.com/questions/72913913/spark-persistent-view-on-a-partition-parquet-file/72914245#72914245CREATE VIEW test as select * from parquet.`/mnt/folder-with-parquet-file(s)/`@Hu...
- 4 kudos
- 1186 Views
- 0 replies
- 0 kudos
BEGINNER
Hey i am beginner and i am intrested to learn data engineering path for azure databrics.Can some one please provide some guidance
- 1186 Views
- 0 replies
- 0 kudos
- 1688 Views
- 1 replies
- 2 kudos
How to get the normalized hours value for the Databricks EMR cluster?
Hi Hubert!I'm working on a usecase to get the compute usage stats. I used boto3 code and describe_cluster() function to get the normalized instance hours value for the EMR cluster. I would like to know if there is an equivalent for this normalized in...
- 1688 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi Bala, nice to meet you.Yes, there is an equivalent. For example, in Databricks, you need to use the get method in cluster API or CLI implementation of API.https://docs.databricks.com/dev-tools/api/latest/clusters.html#get
- 2 kudos
- 18927 Views
- 1 replies
- 2 kudos
Creating permanent views from dataframes?
Hi:It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, somet...
- 18927 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi Kaniz:This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views ca...
- 2 kudos
- 1124 Views
- 0 replies
- 0 kudos
www.mca.gov.in
Today, the entire private limited company registration process and other regulatory filings are paperless; documents are filed electronically through the MCA website and is processed at the Central Registration Centre (CRC). Online Private Limited Co...
- 1124 Views
- 0 replies
- 0 kudos
- 2740 Views
- 2 replies
- 2 kudos
Simba ODBC datetime with millisecond overflows
Hello,Using odbc 2.6.24.1041-2 for Linux, when inserting rows with milliseconds precision date (e.g. 2022-07-03 13:57:48.500) precision I get:2022/07/03 14:41:19 SQLExecute: {22008} [Simba][Support] (40520) Datetime field overflow resulting from inva...
- 2740 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
I was passing a string (e.g. '2022-07-03 13:57:48.500') to the Golang SQL driver which is not working if the ms part is specified, but otherwise it works (e.g. '2022-07-03 13:57:48'). Passing a native Golang time.Time seems to work for timestamps wit...
- 2 kudos
- 1018 Views
- 0 replies
- 5 kudos
Databricks Office Hours Register for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks expe...
Databricks Office HoursRegister for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks experts! Our next events are scheduled for July 13th & July 27th from 8:00am - 9:00am PT | 3:00pm - 4:00pm GM...
- 1018 Views
- 0 replies
- 5 kudos
- 1267 Views
- 0 replies
- 0 kudos
Unexpected behaviour when creating multiple Account-Level Accounts
We've had a qustion regarding possibly unexpected behaviour when creating multiple accounts on the account-level on https://accounts.cloud.databricks.com/.Short Version:It's possible to create multiple accounts with different letter-cases on https://...
- 1267 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
