- 2058 Views
- 3 replies
- 0 kudos
Hi,I am getting data from CDC on SQL Server using Informatica which is writing parquet files to ADLS. I read the parquet files using DLT and end up with the date data as a string such as this'20240603164746563' I couldn't get this to convert using m...
- 2058 Views
- 3 replies
- 0 kudos
Latest Reply
Checking on my current code, this is what I am using, which works for me because we don't use daylight savings time. from_utc_timestamp(date_time_utc, 'UTC-7') as date_time_local
2 More Replies
by
GeKo
• New Contributor III
- 15585 Views
- 5 replies
- 0 kudos
Hello,after switching to "shared cluster" usage a python job is failing with error message: Py4JJavaError: An error occurred while calling o877.load.
: org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User...
- 15585 Views
- 5 replies
- 0 kudos
Latest Reply
Uj337
New Contributor III
Hi @GeKo The checkpoint directory, is that set on cluster level or how do we set that ? Can you please help me with this ?
4 More Replies
- 1194 Views
- 1 replies
- 0 kudos
Hi all.I have a huge data migration project using medallion architecture, UC, notebooks and workflows . One of the relevant requirements we have is to capture all data dependencies (upstreams and downstreams) using data lineage. I've followed all re...
- 1194 Views
- 1 replies
- 0 kudos
Latest Reply
Hello @RobsonNLPT ,
Yes SQL CTE are supported by the data lineage service. You can track table that were created using CTEs.
Here is an example that demonstrate the feature.
CREATE TABLE IF NOT EXISTS
mpelletier.dbdemos.menu (
recipe_id INT,
...
- 776 Views
- 3 replies
- 1 kudos
Hi,I need to ingest and transform historical climate data into a Delta table. The data is stored in .nc format (NetCDF). To work with this format, specific C libraries for Python are required, along with particular versions of Python libraries (e.g.,...
- 776 Views
- 3 replies
- 1 kudos
Latest Reply
Great, please let us know in case any assistance is needed
2 More Replies
- 989 Views
- 5 replies
- 0 kudos
I am trying to automate the creation of git folders in user workspace directories triggered by GitHub feature branch creation. When developers create feature branches in GitHub, we want a service principal to automatically create corresponding git fo...
- 989 Views
- 5 replies
- 0 kudos
Latest Reply
Hi @Brianhourigan,
Can you please DIM your suggestions? I can add it to our internal AHA idea.
4 More Replies
- 1324 Views
- 5 replies
- 0 kudos
Hi All,Is there a way to restore deleted databricks jobs?Thank you.
- 1324 Views
- 5 replies
- 0 kudos
Latest Reply
Hi @iptkrisna ,Currently, there is no option to recover deleted items. In architectures, it not necessary to control or manage the final code available in the system. Instead, the focus should be controlling and managing how code and jobs are deploye...
4 More Replies
- 1395 Views
- 1 replies
- 1 kudos
In our existing notebooks, the scripts are reliant on RDDs. However, with the upgrade to Unity Catalog, RDDs will no longer be supported. We need to explore alternative approaches or tools to replace the use of RDDs. Could you suggest the best practi...
- 1395 Views
- 1 replies
- 1 kudos
Latest Reply
To transition from using RDDs (Resilient Distributed Datasets) to alternative approaches supported by Unity Catalog, you can follow these best practices and migration strategies:
Use DataFrame API: The DataFrame API is the recommended alternative to...
- 970 Views
- 7 replies
- 0 kudos
I'm exploring whether serverless (https://docs.databricks.com/en/jobs/run-serverless-jobs.html#create-a-job-using-serverless-compute) could be useful for our use case. I'd like to see an example of using serverless via the API. The docs say "To learn...
- 970 Views
- 7 replies
- 0 kudos
- 857 Views
- 6 replies
- 0 kudos
At my org, when we start a databricks cluster, it oftens takes awhile to become available (due to (1) instance provisioning, (2) library loading, and (3) init script execution). I'm exploring whether an instance pool could be a viable strategy for im...
- 857 Views
- 6 replies
- 0 kudos
Latest Reply
Sure, I will inform the team in charge of it to review it.
5 More Replies
- 460 Views
- 1 replies
- 1 kudos
Hi everyone,I do not want to use 4 DBU/h XS warehouse since I have very tiny data on the new startup. I want to create a minimal cluster and run it as the underlying SQL engine for my dashboard.Thanks.
- 460 Views
- 1 replies
- 1 kudos
Latest Reply
Unfortunately no, as dashboards are part of the SQL service on the platform they are designed to work with SQL warehouses only, you can create Notebook dashboards that will be able to work with regular clusters but functionalities will be limited in ...
- 805 Views
- 5 replies
- 1 kudos
I have a continuous workflow. It is continuous because I would like it to run every minute and if it has stuff to do the first task will take several minutes. As I understand, continuous workflows won't requeue while a job is currently running, where...
- 805 Views
- 5 replies
- 1 kudos
Latest Reply
Hi @h2p5cq8,
No problem! and you can have the queue option disabled to stop it. Go to the Advanced settings in the Job details side panel and toggle off the Queue option to prevent jobs from being queued
4 More Replies
- 786 Views
- 1 replies
- 0 kudos
Hi, I'm using the Databricks asset bundle to deploy my job to Azure Databricks.I want to configure the Databricks bundle so that when anyone runs the Azure pipeline, a job is created under their name in the format dev_username_job.Using a personal ac...
- 786 Views
- 1 replies
- 0 kudos
Latest Reply
Were you able to solve this?
- 3783 Views
- 5 replies
- 0 kudos
Hi all, I am using the Azure Databricks Microsoft Managed Enterprise Application scope (2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/user_impersonation) to fetch an access token on behalf of a user. The authentication process is successful; however, the acce...
- 3783 Views
- 5 replies
- 0 kudos
Latest Reply
Hi @ahsan_aj,
You can modify your token request by adding a claims parameter
const claimsRequest = {
"access_token": {
"groups": null
}
https://learn.microsoft.com/en-us/security/zero-trust/develop/configure-tokens-gro...
4 More Replies
- 4262 Views
- 2 replies
- 0 kudos
- 4262 Views
- 2 replies
- 0 kudos
Latest Reply
It looks like Databricks making things difficult to use iceberg tables. There is no clear online documentation or steps provided to use with plain spark & spark sql, and the errors thrown in the Databricks environment are very cryptic.They wanted to...
1 More Replies
- 1206 Views
- 1 replies
- 0 kudos
Having problem in connecting to Databrikcs with service principal from tableau . Wanted to how how tableau extracts refreshing connecting to databricks , is it via individual Oauth or service principal
- 1206 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @cheerwthraj,
To connect Tableau to Databricks and refresh extracts, you can use either OAuth or service principal authentication.
For best practices, please refer to the below link,
https://docs.databricks.com/en/partners/bi/tableau.html#best-pr...
