- 1330 Views
- 1 replies
- 0 kudos
How to import .dbc files above size limit?
Above is the screenshot of error, is there any other way of processing dbc files? Do no have access/backup to previous workspace where this code is imported from
- 1330 Views
- 1 replies
- 0 kudos
- 0 kudos
@ByteForge - Kindly raise a support case with Databricks to work with Engg to increase the limits for your workspace.
- 0 kudos
- 1425 Views
- 2 replies
- 0 kudos
Resolved! Is it more performant to run optimize table commands on a serverless SQL warehouse or elsewhere?
Is it more performant to run optimize table commands on a serverless SQL warehouse or in a job or all-purpose compute cluster? I would presume a serverless warehouse would be faster, but I don't know how to test this.
- 1425 Views
- 2 replies
- 0 kudos
- 0 kudos
@ac0 Good day! Serverless SQL warehouses are likely to execute "optimize table" commands faster than job or all-purpose compute clusters due to their rapid startup time, quick upscaling for low latency, and efficient handling of varying query demand....
- 0 kudos
- 1203 Views
- 1 replies
- 0 kudos
how to transform json-stat 2 filte to SparkDataFrame ? how to keep order on MapType structure ?
Hi,I am using different json files of type json-stat2. These kind of json file is quite common used in national statistisc bureau. Its multi dimensional with multy arrays. Using python environment kan we use pyjstat package to easily transform json...
- 1203 Views
- 1 replies
- 0 kudos
- 0 kudos
MapType does not maintain order (json itself too).Can you apply the ordering yourself afterwards?
- 0 kudos
- 1277 Views
- 2 replies
- 0 kudos
cant read json file with just 1,75 MiB ?
Hi,I am realtively new on databricks, although I am conscious about lazy evaluation, transformations and actions and peristence.I have a json file (complex-nested) with about 1,73 MiB. when df = spark.read.option("multiLine", "false").json('dbfs:/mnt...
- 1277 Views
- 2 replies
- 0 kudos
- 0 kudos
This can be resolved by redefining the schema structure explicitly and using that schema to read the file. from pyspark.sql.types import StructType, StructField, StringType, IntegerType, ArrayType# Define the schema according to the JSON structuresch...
- 0 kudos
- 2581 Views
- 4 replies
- 0 kudos
Resolved! performance issues when readingjson-stat2
Hi,I am realtively new on databricks, although I am conscious about lazy evaluation, transformations and actions and peristence.I have a json file (complex-nested) with about 1,73 MiB. when df = spark.read.option("multiLine", "false").json('dbfs:/mnt...
- 2581 Views
- 4 replies
- 0 kudos
- 0 kudos
Please give me a kudos if this works.Efficiency in Data Collection: Using .collect() on large datasets can lead to out-of-memory errors as it collects all rows to the driver node. If the dataset is large, consider alternatives such as extracting only...
- 0 kudos
- 1395 Views
- 2 replies
- 0 kudos
Asset Bundles: Adding project_directory in DBT task breaks previous python task
Hi, I have a job consisting of three tasks: tasks: - task_key: Kinesis_to_S3_new spark_python_task: python_file: ../src/kinesis.py parameters: ["${var.stream_region}", "${var.s3_base_path}"] j...
- 1395 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ajay-Pandey ,thank you for the hints. I will try to recreate the job via UI. I ran the tasks in a Github workflow. The file locations are mixed: the first two tasks (python and dlt) are located in the databricks/src folder. The dbt files come fro...
- 0 kudos
- 2762 Views
- 2 replies
- 1 kudos
Can't import local files under repo
I have a yaml file inside one of the sub dir in Databricks, I have appended the repo path to sys. Still I can't access this file. https://docs.databricks.com/_static/notebooks/files-in-repos.html
- 2762 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @chandan_a_v ,were you able to solve this issue?I am also experiencing the same thing where i cannot move file with extension .yml from repo folder to shared workspace folder.As per documentation, this is the limitation or functionality of data...
- 1 kudos
- 1005 Views
- 1 replies
- 0 kudos
Delta live table is inserting data multiple times
So I have created a delta live table Which uses spark.sql() to execute a query And uses df.write.mode(append).insert intoTo insert data into the respective table And at the end i return a dumy table Since this was the requirement So now I have also ...
- 1005 Views
- 1 replies
- 0 kudos
- 0 kudos
whats your source? your sink is a delta table correct? how do you verify that there are no inserts happening?
- 0 kudos
- 6790 Views
- 6 replies
- 0 kudos
Resolved! Not able to create external table in a schema under a Catalog.
Problem StatementCluster 1 (Shared Cluster) is not able to read the file location at "dbfs:/mnt/landingzone/landingzonecontainer/Inbound/" and hence we are not able to create an external table in a schema inside Enterprise Catalog.Cluster 2 (No Isola...
- 6790 Views
- 6 replies
- 0 kudos
- 0 kudos
External table creation failing with error :- UnityCatalogServiceException:[RequestId=**** ErrorClass=INVALID_PARAMETER_VALUE] Unsupported path operation PATH_CREATE_TABLE on volume.Able to access and create files on external location.
- 0 kudos
- 1534 Views
- 1 replies
- 0 kudos
run md5 using CLI
Hi,I want to run a md5 checksum on the uploaded file to databricks. I can generate md5 on the local file but how do I generate one on uploaded file on databricks using CLI (Command line interface). Any help would be appreciated.I tried running databr...
- 1534 Views
- 1 replies
- 0 kudos
- 7549 Views
- 3 replies
- 1 kudos
Connect Databricks hosted on Azure, with RDS on AWS.
We have Databricks set up and running on Azure. Now we want to connect it with RDS (AWS) to transfer data from RDS to Azure DataLake using the Databricks.I could find the documentation on how to do it within the same cloud (Either AWS or Azure) but n...
- 7549 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Danial Malik​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
- 1 kudos
- 1509 Views
- 0 replies
- 0 kudos
Capture Automatically Added tags
Can we capture automatically added tags (ex: RunName) from azure data bricks job cluster to parameters or custom tags in azure data factory
- 1509 Views
- 0 replies
- 0 kudos
- 2630 Views
- 4 replies
- 0 kudos
Shallow clone and issue with MODIFY permission to source table
Hi,I'm running shallow clone for external delta tables. The shallow clone is failing for source tables where I don't have MODIFY permission. I'm getting below exception. I don't understand why MODIFY permission to source table is required. Is there a...
- 2630 Views
- 4 replies
- 0 kudos
- 0 kudos
Also check this documentation on access mode :Shallow clone for Unity Catalog tables | Databricks on AWS Working with Unity Catalog shallow clones in Single User access mode, you must have permissions on the resources for the cloned table source as w...
- 0 kudos
- 1925 Views
- 1 replies
- 0 kudos
Fixed interval micro-batches and AvailableNow Trigger
What is the fundamental difference between Fixed interval micro-batches and AvailableNow Trigger, given that both can consume in micro-batch based on the desire size of the micro batch ? Is the fundamental difference the fact that AvailableNow shut d...
- 1925 Views
- 1 replies
- 0 kudos
- 7300 Views
- 9 replies
- 0 kudos
Pulsar Streaming (Read) - Benchmarking Information
We are doing a first time implementation of data streaming reading from a partitioned pulsar topics to a delta table managed by UC. We are unable to scale the job beyond about ~ 40k msgs/sec. Beyond 40k msgs/sec , the job fails. I'd imagine Databric...
- 7300 Views
- 9 replies
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
