- 613 Views
- 0 replies
- 0 kudos
The summit was great #Summit22​Â
The summit was great #Summit22​
- 613 Views
- 0 replies
- 0 kudos
Forum Posts
- 2074 Views
- 1 replies
- 0 kudos
Is there a way to watch a forum post ?
Is there a way to watch a forum post ? I cant see one.
- 2074 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Thanks, I was just wondering if there was a watch with notification function.
- 0 kudos
- 3626 Views
- 4 replies
- 0 kudos
Resolved! Add custom tags to DLT cluster
We have a policy in our AWS account that whenever an EC2 instance is created, there are 5 mandatory tags that need to be added. When we create an All-Purpose cluster in the Databricks console, we can easily add these under Advance Options > Tags. How...
- 3626 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
@Tomasz Bacewicz​ Perfect!! worked like a charm...thank you!!do you think auto loader or DLT is a better approach to implementing a CDC pipeline in Databricks?
- 0 kudos
- 7560 Views
- 3 replies
- 3 kudos
Delta Live Table Pipeline Load Error
I am getting this error when running Delta Live Table Pipelines pulling from a source in our sandbox's local dbfs folder. It says the user is not authorized to perform this operation, whereas I am able to see the data when I run a simple select state...
- 7560 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
Hey there @Abhay Sudhakaran​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution and mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...
- 3 kudos
- 3580 Views
- 3 replies
- 8 kudos
Resolved! AWS Private Link for DB workspace
What's the best way to set up a private link connection to Databricks without loosing the current settins? workspace
- 3580 Views
- 3 replies
- 8 kudos
Latest Reply
- 8 kudos
At the data and AI summit, there was a nice talk about AWS private link. Here is a link if you have access:https://dataaisummit.com/session-virtual/?v2477da705118cc74fd14460db021e1784e2eed5a7982c6482ec95cb2e86d259644b8741959f52a49e0e6908b82a9d860=C22...
- 8 kudos
- 4239 Views
- 3 replies
- 3 kudos
Resolved! How do I setup the right cluster?
- 4239 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
For general cluster decision making refer to this article https://docs.microsoft.com/en-gb/azure/databricks/clusters/cluster-config-best-practicesOnce you've selected a cluster that makes sense, run it and check your ganglia metrics to see whether yo...
- 3 kudos
- 4824 Views
- 5 replies
- 2 kudos
Resolved! DLT Pipeline and Job Cluster
We have written few python functions(methods within a class) and packaged them as a wheel library.In the as-is situation we use to install that wheel library in All-Purpose cluster that we already have created. It works fine.In the to-be situtation(D...
- 4824 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
Does it give you an error when running the DLT pipeline specifically on the %pip command or does it not work in some other way? If it's the former, could you share the path format that you're using for the %pip command path?
- 2 kudos
- 7103 Views
- 3 replies
- 4 kudos
Resolved! Spark persistent view on a partition parquet file
In Spark, is it possible to create a persistent view on a partitioned parquet file in Azure BLOB? The view must be available when the cluster restarted, without having to re-create that view, hence it cannot be a temp view.I can create a temp view, b...
- 7103 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
Here is what worked for me. Hope this helps someone else: https://stackoverflow.com/questions/72913913/spark-persistent-view-on-a-partition-parquet-file/72914245#72914245CREATE VIEW test as select * from parquet.`/mnt/folder-with-parquet-file(s)/`@Hu...
- 4 kudos
- 1402 Views
- 0 replies
- 0 kudos
BEGINNER
Hey i am beginner and i am intrested to learn data engineering path for azure databrics.Can some one please provide some guidance
- 1402 Views
- 0 replies
- 0 kudos
- 2364 Views
- 1 replies
- 2 kudos
How to get the normalized hours value for the Databricks EMR cluster?
Hi Hubert!I'm working on a usecase to get the compute usage stats. I used boto3 code and describe_cluster() function to get the normalized instance hours value for the EMR cluster. I would like to know if there is an equivalent for this normalized in...
- 2364 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi Bala, nice to meet you.Yes, there is an equivalent. For example, in Databricks, you need to use the get method in cluster API or CLI implementation of API.https://docs.databricks.com/dev-tools/api/latest/clusters.html#get
- 2 kudos
- 22907 Views
- 1 replies
- 2 kudos
Creating permanent views from dataframes?
Hi:It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, somet...
- 22907 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi Kaniz:This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views ca...
- 2 kudos
- 3503 Views
- 2 replies
- 2 kudos
Simba ODBC datetime with millisecond overflows
Hello,Using odbc 2.6.24.1041-2 for Linux, when inserting rows with milliseconds precision date (e.g. 2022-07-03 13:57:48.500) precision I get:2022/07/03 14:41:19 SQLExecute: {22008} [Simba][Support] (40520) Datetime field overflow resulting from inva...
- 3503 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
I was passing a string (e.g. '2022-07-03 13:57:48.500') to the Golang SQL driver which is not working if the ms part is specified, but otherwise it works (e.g. '2022-07-03 13:57:48'). Passing a native Golang time.Time seems to work for timestamps wit...
- 2 kudos
- 1277 Views
- 0 replies
- 5 kudos
Databricks Office Hours Register for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks expe...
Databricks Office HoursRegister for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks experts! Our next events are scheduled for July 13th & July 27th from 8:00am - 9:00am PT | 3:00pm - 4:00pm GM...
- 1277 Views
- 0 replies
- 5 kudos
- 1870 Views
- 0 replies
- 0 kudos
Unexpected behaviour when creating multiple Account-Level Accounts
We've had a qustion regarding possibly unexpected behaviour when creating multiple accounts on the account-level on https://accounts.cloud.databricks.com/.Short Version:It's possible to create multiple accounts with different letter-cases on https://...
- 1870 Views
- 0 replies
- 0 kudos
- 5315 Views
- 3 replies
- 1 kudos
Resolved! ipywidgets not working in DBR 11.0 on Community Edition
I'm looking forward to using ipywidgets which should be working in DBR 11.0 as they provide more options when creating a notebook UI. I saw that DBR 11.0 is available as of yesterday so I created a test cluster in the Databricks Community Edition ju...
- 5315 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Wayne Deleersnyder​ I was able to import ipywidgets in DBR 11.0. As you can see in the output below. The slider is visibleYou are facing this issue probably because the community edition has limited access. To get all the features you should at l...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
