- 6011 Views
- 5 replies
- 0 kudos
On-behalf-of token creation for service principals is not enabled for this workspace
Hi AllI just wanted to create PAT for Databricks Service Principle but getting below code while hitting API or using CLI - Please help me to create PAT for the same.#dataengineering #databricks
- 6011 Views
- 5 replies
- 0 kudos
- 0 kudos
You can generate the token while logged in as the Service Principle via the Azure CLI in a Command Prompt window. To do so, make sure to install the Azure CLI and the Databricks CLI with it.Install the Azure CLI for Windows | Microsoft LearnInstall ...
- 0 kudos
- 2825 Views
- 1 replies
- 0 kudos
Reading CloudWatch Logs from AWS Kinesis
If you have AWS CloudWatch subscribed to write out logs to AWS Kinesis, the Kinesis stream is base64 encoded and the CloudWatch logs are GZIP compressed. The challenge we faced was how to address that in pyspark to be able to read the data. We were ...
- 2825 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you so much, this is very helpful
- 0 kudos
- 3225 Views
- 3 replies
- 3 kudos
Resolved! What is the difference between spark inferschema and cloudFiles.inferColumnTypes?
We have been using spark.read with inferSchema = True to validate AutoLoader schema inferencing. But I have a suspicion that they do these differently from each other and may not always yield the identical results.Has anyone ever answered this questi...
- 3225 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @BF7 Yes — there is a difference between how spark.read(...).option("inferSchema", "true")and Auto Loader's schema inference (cloudFiles.schemaHints, cloudFiles.inferColumnTypes, etc.) work.They are not guaranteed to produce identical results,Key ...
- 3 kudos
- 2220 Views
- 3 replies
- 1 kudos
Resolved! springml sftp with spark 3.x
Is there a version of springml spark-sftp that works with spark 3.x and scala 2.12? If so can you point me to it or how to load it in my compute?
- 2220 Views
- 3 replies
- 1 kudos
- 1 kudos
For Python you might want to look at Paramiko, it seems that it might be an option. You could also look at ETL tools like Airbyte, Rivery, CData, etc.
- 1 kudos
- 10480 Views
- 3 replies
- 0 kudos
SQLSTATE: XX000 The Spark SQL phase planning failed with an internal error.
Hello everyone,I am currently working with a SQL Warehouse and have been getting the following error message:[INTERNAL_ERROR ] The Spark SQL phase planning failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, re...
- 10480 Views
- 3 replies
- 0 kudos
- 0 kudos
I have tried to simplify the query as much as possible to see if that helps but the bug still persists. The problem should be something with the way Databricks treats columns passed as arguments for a function.I tried these queries:select * FROM VALU...
- 0 kudos
- 12785 Views
- 15 replies
- 3 kudos
API for Restarting Individual Failed Tasks within a Job?
Hi everyone,I'm exploring ways to streamline my workflow in Databricks and could really use some expert advice. In my current setup, I have a job (named job_silver) with multiple tasks (e.g., task 1, task 2, task 3). When one of these tasks fails—say...
- 12785 Views
- 15 replies
- 3 kudos
- 3 kudos
Hey @minhhung0507 - quick question - what is the cluster type you're using to run your workflow?I'm using a shared, interactive cluster, so I'm passing the parameter {'existing_cluster_id' : task['existing_cluster_id']}in the payload. This parameter ...
- 3 kudos
- 2555 Views
- 4 replies
- 0 kudos
Resolved! Debugging jobs/run-now endpoint
I am not being able to run jobs/runnow endpoint. I am getting an error asError fetching files: 403 - {"error_code":"PERMISSION_DENIED","message":"User xxxx-dxxxx-xxx-xxxx does not have Manage Run or Owner or Admin permissions on job 437174060919465",...
- 2555 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @smpa01 - The PAT you're using belongs to Service Principal or your personal token? If SP, it should have permissions to run the DBX workflow. Let me know if any questions.
- 0 kudos
- 3575 Views
- 5 replies
- 5 kudos
Resolved! Dlt pipeline showing legacy , even though all things are latest
Some of old dlt pipelines in my databricks workspace are showing legacy,I am using serverless pipeline with mode - preview.Anything which I missed ?
- 3575 Views
- 5 replies
- 5 kudos
- 1218 Views
- 1 replies
- 0 kudos
Writing files using multithreading to dbfs
Hello,I am reading in xml files from AWS S3 and storing them on dbfs:/ using multithreaded code. The code itself seems to be fine as for the first +- 100 000 files it works without issues and I can see the data arriving on DBFS.However it will always...
- 1218 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @daan_dw I think this issue mainly comes from using multithreading to handle XML files while interacting with both S3 and DBFS. When the thread count gets too high, it likely causes race conditions.To avoid this:Try reducing the number of threads....
- 0 kudos
- 2466 Views
- 2 replies
- 2 kudos
Resolved! How do you think continuing to use instance profile to S3 multi part upload?
My team is currently using an instance profile to upload data to S3 since we only have Hive Metastore.I like Unity Catalog a lot, but my code uses multipart upload to S3 for efficiency.https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview...
- 2466 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @lingareddy_Alva ,Thank you for your excellent response. I really appreciated it.I couldn't find the mention that says "Instance profiles are still supported but should be used for specific, advanced access cases." I will use it for now, recognizi...
- 2 kudos
- 2320 Views
- 3 replies
- 0 kudos
How to set DLT pipeline warning alert?
Hi there,The example description of custom event hooks in the documentation is not clear enough, I do not know how to implement it inside python functions. event-hooks My Code: %python # Read the insertion of data raw_user_delta_streaming=spark.rea...
- 2320 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @zmsoft The event hook provided, user_event_hook, must be a Python callable that accepts exactly one parameter - a dictionary representation of the event that triggered the execution of this event hook. The return value of the event hook has no s...
- 0 kudos
- 2344 Views
- 2 replies
- 2 kudos
Resolved! Adding column masks to a column using the DLT Python create_streaming_table API
I'm having difficulty adding a mask function to columns while creating streaming tables using the DLT Python method create_streaming_table() like this but it does not work, the streaming table is created but no column is masked:def prepare_column_pro...
- 2344 Views
- 2 replies
- 2 kudos
- 2 kudos
@NamNguyenCypher Delta Live Tables’ Python API does not currently honor column-mask metadata embedded in a PySpark StructType. Masking (and row filters) on DLT tables are only applied when you define your table with a DDL-style schema that includes a...
- 2 kudos
- 2645 Views
- 5 replies
- 1 kudos
Unexpected SKU Names in Usage Table for Job Cost Calculation
I'm trying to calculate the cost of a job using the usage and list_prices system tables, but I'm encountering some unexpected behavior that I can't explain.When I run a job using a shared cluster, the sku_name in the usage table is PREMIUM_JOBS_SERVE...
- 2645 Views
- 5 replies
- 1 kudos
- 1 kudos
Thank you all for your replies. @lingareddy_Alva what about @Walter_C and @mnorland mentioned about enabling serverless tasks. Is this possible and how?
- 1 kudos
- 1462 Views
- 1 replies
- 0 kudos
DLT Publish event log to metastore
Hi Guys,I am trying to use DLT Publish event log to metastore feature.and I noticed it creates a table with the logs for each DLT pipelines separately. Does it mean it maintains the separate log table for all the DLT tables ( in our case, we have 100...
- 1462 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ankit001mittal Yes, you're right, when you enable the "Publish Event Log to Metastore" option for DLT pipelines, Databricks creates a separate event log table for each pipeline. So, if you have thousands of pipelines, you'll see thousands of log ...
- 0 kudos
- 2481 Views
- 2 replies
- 0 kudos
Repairing running workflow with few failed child jobs
I have a parent job that calls multiple child jobs in workflow, Out of 10 child jobs, one has failed and rest 9 are still running, I want to repair the failed child tasks. can I do that while the other child jobs are running?
- 2481 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi holychs,How are you doing today?, As per my understanding, yes, in Databricks Workflows, if you're running a multi-task job (like your parent job triggering multiple child tasks), you can repair only the failed task without restarting the entire j...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
