- 4780 Views
- 4 replies
- 0 kudos
Databricks XML - Bypassing rootTag and rowTag
I see the current conversion of dataframe to xml need to be improved.My dataframe schema is a perfect nested schema based on structs but when I create a xml I have the follow issues:1) I can't add elements to root2) rootTag and rowTag are requiredIn ...
- 4780 Views
- 4 replies
- 0 kudos
- 0 kudos
Here is one of the ways to use the struct field name as rowTag: import org.apache.spark.sql.types._ val schema = new StructType().add("Record", new StructType().add("age", IntegerType).add("name", StringType)) val data = Seq(Row(Row(18, "John ...
- 0 kudos
- 4101 Views
- 3 replies
- 5 kudos
The risks of code execution by default on widget change
Taking from my experience, the default action of widgets triggering code execution upon value change poses risks that outweigh the convenience in certain scenarios. While this feature may seem advantageous in some cases, it can lead to unintended con...
- 4101 Views
- 3 replies
- 5 kudos
- 5 kudos
I definitely have to agree with the original point- if you have a notebook that you import, and you touch any widget value you're running code, most likely accidentally. I'd love to see a workspace or user type option where you can change the default...
- 5 kudos
- 2492 Views
- 2 replies
- 1 kudos
databricks spark XML Writer
Hi.I'm trying to generate XML as output base on my nested dataframe. Everything is ok except by I don't know how to add elements to rootTag.I can add elements from rowtag but not in rootTag. Same problems to add attributes to root <books version = "...
- 2492 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @RobsonNLPT ,Thanks for bringing up your concerns, always happy to help Can you please refer to the below document to read and write the XML files? https://docs.databricks.com/en/query/formats/xml.html Please let me know if this helps and leave a...
- 1 kudos
- 3088 Views
- 1 replies
- 0 kudos
FileAlreadyExistsException error while analyzing table in Notebook
Databricks experts, I'm new to Databricks, and encounter an issue with the ANALYZE TABLE command in the Notebook. I created two tables nyc_taxi and nyc_taxi2, from one csv file.When executing the following command in Notebook, analyze table nyc_taxi2...
- 3088 Views
- 1 replies
- 0 kudos
- 11414 Views
- 4 replies
- 0 kudos
Running an exe file in databricks
Hello I have an executable file which i want to host and run from databricks. is this possible in databricks using DBFS ?If NOT what are the other ways to it in databricks ?
- 11414 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello, I don't have much information on what kind of executables you would like to run in databricks however, I can think of two solutions : Solution 1: Deploy your code in azure container registry as an image and use the endpoint in data bricks. Sol...
- 0 kudos
- 3814 Views
- 1 replies
- 0 kudos
JWT Encoding error while using Azure secret key
My secret value in Azure key vault is like below.private_key="""-----BEGIN RSA PRIVATE KEY-----********-----END RSA PRIVATE KEY-----"""Running this command in Databricks notebook - jwt.encode(claim_set,private_key,algorithm='RS256')While using the ab...
- 3814 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks much for your troubleshooting methods.Validated the secret scopes, accessing secrets. These looks fine.Key format - I feel problem is with the key format only. As of now I'm awaiting on Azure subscription access. But I printed the secret value...
- 0 kudos
- 10396 Views
- 3 replies
- 1 kudos
stop autoloader with continuous trigger programatically
Hi,I am running autoloader with continuous trigger. How can I stop this trigger during some specific time, only if no data pending and current batch process is complete. How to check how many records pending in queue and current state.Regards,Sanjay
- 10396 Views
- 3 replies
- 1 kudos
- 26429 Views
- 2 replies
- 1 kudos
Resolved! Deny assignment modification to allow attach/detach of disks in azure databricks
Our application does storage autoscaling on Azure. We would like to deploy our solution with Azure databricks. But even though the service principal associated with our application has the necessary roles and permissions to attach/detach a disk from ...
- 26429 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you for your replyIs there any way databricks provides to bypass the deny assignment for specific apps? I noticed in the deny assignment unity-catalog-access-connector has been provided exlusion under the excludePrincipals section. is there a w...
- 1 kudos
- 1752 Views
- 1 replies
- 0 kudos
Create persistent Scala UDF
I've created a UDF in Scala (using Databricks notebook) which checks if the value of a field in a table is greater than 0, then sum those values. UDF is working as expected. The problem is, that this UDF will be used by multiple SQLs across different...
- 1752 Views
- 1 replies
- 0 kudos
- 0 kudos
In the child notebook:def my_udf(x): return x + 1 spark.udf.register("my_udf", my_udf) dbutils.notebook.exit(my_udf)In the master notebook:child_udf = dbutils.notebook.run("PathToChildnotebook", timeout_seconds=600) spark.udf.register("my_udf",...
- 0 kudos
- 4152 Views
- 3 replies
- 1 kudos
[UNBOUND_SQL_PARAMETER] When running Placeholder query
I am using the databricks-sql-go library version 1.5.2. I am trying to run a query with placeholders of type '?'The query looks like params, args := databricksParams(values)sql := fmt.Sprintf(`SELECT COUNT(*) FROM %s.%s WHERE %s IN (%s)`, schema, tab...
- 4152 Views
- 3 replies
- 1 kudos
- 1 kudos
Can you print out an example after the Sprintf substitutions?It seems you generated a query with a named parameter: ":_58" But args (which should be a Map) does not have a key named "_58".
- 1 kudos
- 2664 Views
- 1 replies
- 0 kudos
DLT: Only STREAMING tables can have multiple queries.
I am trying to to do a one-time back-fill on a DLT table following the example here: dlt.table() def test(): # providing a starting version return (spark.readStream.format("delta") .option("readChangeFeed", "true") .option("...
- 2664 Views
- 1 replies
- 0 kudos
- 0 kudos
I should also add that when I drop the `backfill` function, validation happens successfully and we get the following pipeline DAG:
- 0 kudos
- 13664 Views
- 1 replies
- 1 kudos
Introducing AI Model Sharing with Databricks!
Today, we're excited to announce that AI model sharing is available in both Databricks Delta Sharing and on the Databricks Marketplace. With Delta Sharing you can now easily share and serve AI models securely within your organization or externally ac...
- 13664 Views
- 1 replies
- 1 kudos
- 1 kudos
I'm eager to dive in and leverage these new features to elevate my AI game with Databricks.This is Johnson from KBS Technologies.Thanks for your update.
- 1 kudos
- 4115 Views
- 3 replies
- 0 kudos
Cannot create a repo because the parent path does not exist
I tried following this docWork With Large Monorepos With Sparse Checkout Support in Databricks Repos | Databricks BlogWhen I hook it up to my repos using Azure DevOps Services and check mark Sparse checkout mode, I pass in a subdirectory in my Cone p...
- 4115 Views
- 3 replies
- 0 kudos
- 11431 Views
- 2 replies
- 0 kudos
Resolved! Show Existing Header From CSV I External Table
Hello, is there a way to load csv data into an external table without the _c0, _c1 columns showing?
- 11431 Views
- 2 replies
- 0 kudos
- 4418 Views
- 3 replies
- 0 kudos
Resolved! Unable to load csv data with correct header values in External tables
Hello, is there a way to load "CSV" data into an external table without the _c0, _c1 columns showing?I've tried using the options within the sql statement that does not appear to work.Which results in this table
- 4418 Views
- 3 replies
- 0 kudos
- 0 kudos
you need set "USING data_source"https://community.databricks.com/t5/data-engineering/create-external-table-using-multiple-paths-locations/td-p/44042
- 0 kudos
