- 4619 Views
- 1 replies
- 0 kudos
Postman Collection Alternatives for Data-Centric API Management?
I’ve been using Postman collections to manage APIs in my data projects, but I’m exploring alternatives. Are there tools like Apidog or Insomnia that perform better for API management, particularly when working with large data sets or data-driven work...
- 4619 Views
- 1 replies
- 0 kudos
- 0 kudos
Insomnia: Insomnia is another strong alternative that is frequently recommended. It is known for its simplicity and effectiveness in making REST API requests. Insomnia supports the import of Postman collections and is praised for its performance and ...
- 0 kudos
- 1041 Views
- 2 replies
- 0 kudos
Is Limit input rate Docs not correct?
In databricks docs it says "If you use maxBytesPerTrigger in conjunction with maxFilesPerTrigger, the micro-batch processes data until either the maxFilesPerTrigger or maxBytesPerTrigger limit is reached."But based on the source code this is not true...
- 1041 Views
- 2 replies
- 0 kudos
- 0 kudos
In my opinion, the reason for not using both options simultaneously is that the framework would face a logical conflict:Should it stop reading after the maximum number of files is reached, even if the size limit hasn’t been exceeded?OrShould it stop ...
- 0 kudos
- 10014 Views
- 1 replies
- 3 kudos
Best Practices for Copying Data Between Environments
Hi Everyone,I'd like to start a discussion about the best practices for copying data between environments. Here's a typical setup:Environment Setup:The same region and metastore (Unity Catalog) is used across environments.Each environment has a singl...
- 10014 Views
- 1 replies
- 3 kudos
- 3 kudos
Using CTAS (CREATE TABLE AS SELECT) might be a more robust solution for your use case: Independence: CTAS creates a new, independent copy of the data, avoiding dependencies on the source tableSimplified access control: Access rights can be managed so...
- 3 kudos
- 5915 Views
- 2 replies
- 1 kudos
Auto Loader changes the order of columns when infering JSON schema (sorted lexicographically)
We are using Auto Loader to read json files from S3 and ingest data into the bronze layer. But it seems auto loader struggles with schema inference and instead of preserving the order of columns from the JSON files, it sorts them lexicographically.Fo...
- 5915 Views
- 2 replies
- 1 kudos
- 1 kudos
Auto Loader's default behavior of sorting columns lexicographically during schema inference is indeed a limitation when preserving the original order of JSON fields is important. Unfortunately, there isn't a built-in option in Auto Loader to maintain...
- 1 kudos
- 4521 Views
- 1 replies
- 0 kudos
Runtime increases exponentially from 11.3 to 13.3
Hello. I am using R on databricks and using the below approach. My Spark version:Single node: i3.2xlarge · On-demand · DBR: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) · us-east-1a, the job takes 1 hourI install all R packages (including a geo...
- 4521 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello! It's possible that the increase in runtime when upgrading from Spark 3.3.0 (DBR 11.3) to Spark 3.4.1 (DBR 13.3) is due to changes in the underlying R runtime or package versions. When you upgrade to a new version of Spark, the R packages that ...
- 0 kudos
- 4994 Views
- 1 replies
- 1 kudos
Changing a Delta Live Table's schema
I have a Delta Live Table whose source is a Kafka stream. One of the columns is a Decimal and I need to change its precision.What's the correct approach to changing the DLT's schema?Just changing the column's precision in the DLT definition will resu...
- 4994 Views
- 1 replies
- 1 kudos
- 1 kudos
To change the precision of a Decimal column in a Delta Live Table (DLT) with a Kafka stream source, you can follow these steps: 1. Create a new column in the DLT with the desired precision.2. Copy the data from the old column to the new column.3. Dro...
- 1 kudos
- 4754 Views
- 1 replies
- 0 kudos
sampleBy stream in DLT
I would like to create a sampleBy (stratified version of sample) copy/clone of my delta table. Ideally, I'd like to do this using a DLT. My source table grows incrementally each month as batch files are added and autoloader picks them up. Id...
- 4754 Views
- 1 replies
- 0 kudos
- 0 kudos
You can create a stratified sample of your delta table using the `sampleBy` function in Databricks. However, DLT does not support the `sampleBy` function directly. To work around this, you can create a notebook that uses the `sampleBy` function to c...
- 0 kudos
- 2285 Views
- 4 replies
- 1 kudos
Resolved! Managed Tables on Azure databricks
Hi everyone,I was trying to understand: when a managed table is created, Databricks stores the metadata in the Hive metastore and the data in the cloud storage managed by it, which in the case of Azure Databricks will be an Azure Storage Account. But...
- 2285 Views
- 4 replies
- 1 kudos
- 910 Views
- 1 replies
- 1 kudos
Resolved! How to send BINARY parameters using the REST Sql API?
We are trying to send a SQL query to the REST API including a BINARY parameter, EX:"INSERT INTO MyTable (BinaryField) VALUES(:binaryData)"We tried to encode the parameter as base64 and specify that is a BINARY type but it throws a mapping error, if w...
- 910 Views
- 1 replies
- 1 kudos
- 1 kudos
Trying to serialize as binary can be pretty challenging, here's a way to do this with base64 - the trick is to serialize as base64 string and insert as binary with unbase64. databricks api post /api/2.0/sql/statements --json '{ "warehouse_id": "wa...
- 1 kudos
- 1446 Views
- 2 replies
- 0 kudos
Alerts for Failed Queries in Databricks
How can we set up automated alerts to notify us when queries executed by a specific service principal fail in Databricks?
- 1446 Views
- 2 replies
- 0 kudos
- 0 kudos
@Alberto_UmanaOur service principal uses the SQL Statement API to execute queries. We want to receive notifications for each query failure. While SQL Alerts are an option, they do not provide immediate responses. Is there a better solution to achieve...
- 0 kudos
- 1602 Views
- 5 replies
- 0 kudos
Resolved! Using FROM_CSV giving unexpected results
Hello, I am trying to use from_csv in the sql warehouse, but I am getting unexpected results:As a small example I am running: WITH your_table AS ( SELECT 'a,b,c\n1,"hello, world",3.14\n2,"goodbye, world",2.71' AS csv_column ) SELECT from_csv(csv_c...
- 1602 Views
- 5 replies
- 0 kudos
- 0 kudos
@seanstachff Here is the code I used to produce the results shown in the image I shared earlier. It's a bit verbose, so I’m not entirely satisfied with it, but I hope it might provide some helpful insights for you.%sql WITH your_table AS ( -- Examp...
- 0 kudos
- 4631 Views
- 1 replies
- 0 kudos
Networking Setup in Standard Tier – VNet Integration and Proxy Issues
Hi everyone,We are working on an order forecasting model using azure databricks and an ml model from Hugging Face and are running into an issue where the connection over SSL (port 443) fails during the handshake (EOF Error SSL 992). We suspect that a...
- 4631 Views
- 1 replies
- 0 kudos
- 0 kudos
It may depend on your UDR setup. If you have a UDR rule routing the traffic to any firewall appliance, it may possibly be related to traffic not being allowed in the firewall. If there is no UDR or UDR rule routes this traffic to the Internet, it wou...
- 0 kudos
- 6816 Views
- 9 replies
- 1 kudos
Switching to File Notification Mode with ADLS Gen2 - Encountering StorageException
Hello,We are currently utilizing an autoloader with file listing mode for a stream, which is experiencing significant latency due to the non-incremental naming of files in the directory—a condition that cannot be altered.In an effort to mitigate this...
- 6816 Views
- 9 replies
- 1 kudos
- 1 kudos
You should also reevaluate your use of premium storage for your landing area files. Typically, storage for raw files does not need to be the fastest and most resilient and expensive tier. Unless you have a compelling reason for premium storage for la...
- 1 kudos
- 970 Views
- 1 replies
- 0 kudos
Databricks Clean Rooms creation
I am trying to create a Databricks Clean Rooms instance, I have been following the video from Databricks youtube channel.As I only have one workspace, to create a clean rooms I have added my own Clean Room sharing identifier,when I do that I get the ...
- 970 Views
- 1 replies
- 0 kudos
- 0 kudos
@angelop I tried it as well and encountered the same error. A new collaborator needs to be set up. If that’s not feasible, it would be advisable to reach out to Databricks support.By the way, the following video provides a more detailed explanation a...
- 0 kudos
- 1988 Views
- 1 replies
- 0 kudos
Autoloader issue
I'm trying to ingest data from Parquet files using Autoloader. Now, I have my custom schema, I don't want to infer the schema from the parquet files.During readstream everything is fine. But during writestream, it is somehow inferring the schema from...
- 1988 Views
- 1 replies
- 0 kudos
- 0 kudos
In this case, please make sure you specify the schema explicitly when reading the Parquet files and do not specify any inference options. Something like spark.readStream.format("cloudFiles").schema(schema)... If you want to more easily grab the schem...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
