- 3129 Views
- 1 replies
- 1 kudos
groupBy without aggregation (Pyspark API)
Hi guys,You have any idea how can I do a groupBy without aggregation (Pyspark API)like: df.groupBy('field1', 'field2', 'field3') My target is make a group but in this case is not necessary count records or aggregationThank you
- 3129 Views
- 1 replies
- 1 kudos
- 1 kudos
df.select("field1","field2","field3").distinct()do you mean get distinct rows for selected column?
- 1 kudos
- 1231 Views
- 0 replies
- 0 kudos
Parse nested json for building footprints
Looking for some help. If anyone has worked with nested json file in Databricks notebook. I am trying to parse nested json file to get coordinates and use that to create polygon for footprint. Do I need to read it as txt? How can I use the Databricks...
- 1231 Views
- 0 replies
- 0 kudos
- 1677 Views
- 1 replies
- 0 kudos
I am trying to read nested data from json file to put it into streaming table using dlt
So i have this nested data with more than 200+columns and i have extracted this data into json file when i use the below code to read the json files, if in data there are few columns which have no value at all it doest inclued those columns in schema...
- 1677 Views
- 1 replies
- 0 kudos
- 0 kudos
replying to my above questionwe cannot use inferschema on streaming table we need to externally specify schema can anyone please suggest a way to write data in nested form to streaming table and if this is possible?
- 0 kudos
- 3968 Views
- 3 replies
- 1 kudos
How to connect Analysis Service Cube with Databricks notebook
I am trying to connect AS Cube with Databricks notebook but unfortunately didn't find any solution yet. is there any possible way to connect AS cube with databricks notebook? if yes can someone please guide me
- 3968 Views
- 3 replies
- 1 kudos
- 1 kudos
I am able to connect Azure analysis services using Azure Analysis services rest api. is yours on-prem?
- 1 kudos
- 5479 Views
- 4 replies
- 5 kudos
Resolved! REST API for Stream Monitoring
Hi, everyone. I just recently started using Databricks on Azure so my question is probably very basic but I am really stuck right now.I need to capture some streaming metrics (number of input rows and their time) so I tried using the Spark Rest Api ...
- 5479 Views
- 4 replies
- 5 kudos
- 5 kudos
hi @Roberto Baldrez​ ,if you think that @Gaurav Rupnar​ solved your question, then please select it as best response to it can be moved to the top of the topic and it will help more users in the future.Thank you
- 5 kudos
- 3737 Views
- 2 replies
- 2 kudos
I have created a DLT pipeline which reads data from json files which are stored in databricks volum
I have created a DLT pipeline which reads data from json files which are stored in databricks volume and puts data into streaming table This was working fine.when i tried to read the data that is inserted into the table and compare the values with t...
- 3737 Views
- 2 replies
- 2 kudos
- 2 kudos
Keep your DLT code separate from your comparison code, and run your comparison code once your DLT data has been ingested.
- 2 kudos
- 1920 Views
- 1 replies
- 1 kudos
Unity Catalog Migration
Hello,We are in the process of migrating to Unity Catalog. So, can I know how to automate the process of Refactoring the Notebooks to Unity Catalog.
- 1920 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Avinash_Narala There is no one-click solution to refactor all table names notebooks with UC's three level namespaces. At least, manual updating table names is required during the migration process.One option is you can you search feature. Search ...
- 1 kudos
- 10004 Views
- 3 replies
- 0 kudos
Disable Machine Learning and Job Creation
We are working on creating a new databricks workspace for external entities. We have disabled Cluster and Warehouse creation permission but the external users are still able to create Jobs and job clusters. Is there a way to revoke Job creation permi...
- 10004 Views
- 3 replies
- 0 kudos
- 0 kudos
It permits cluster creation during Workflow/Job/DLT pipeline creation. However, when attempting to start any of these, it fails with a 'Not authorized to create compute' error. Please try it and inform me of the outcome
- 0 kudos
- 1293 Views
- 1 replies
- 0 kudos
duplicate files in delta table
I am facing this issue from long time but so far there is no solution. I have delta table. My bronze layer is picking up the old files (mostly 8 days old file) randomly. My source of files is azure blob storage.
- 1293 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @jaimeperry12345 I will need more information to direct you in the right direction: Confirm the behavior: Double-check that your Delta table is indeed reading 8-day-old files randomly. Provide any logs or error messages you have regarding this.Ex...
- 0 kudos
- 6920 Views
- 3 replies
- 1 kudos
Resolved! can I use volume for external table location?
Hello,I have a parquet file test.parquet in the volume volume_ext_test. Tried to create an external table as below, it failed and says it "is not a valid URI".create table catalog_managed.schema_test.tbl_vol asselect * from parquet.`/Volumes/catalog_...
- 6920 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @PaulineX As per the documentation, you cannot use volume for storing table data. It's for loading, storing and accessing files. You cannot use volumes as a location for tables. Volumes are intended for path-based data access only. Use tables for ...
- 1 kudos
- 2399 Views
- 0 replies
- 0 kudos
Setting Column Only on Insert with DLT's apply_changes CDC Merge
Hi Databricks Community,We've encountered an issue with setting a column only on insert when using DLT's apply_changes CDC merge functionality. It's important to note that this capability is available when using the regular Delta merge operation, spe...
- 2399 Views
- 0 replies
- 0 kudos
- 2104 Views
- 1 replies
- 0 kudos
Migrating Job Orchestration to Shared Compute and avoiding(?) refactoring
In an effort to migrate our data objects to the Unity Catalog, we must migrate our Job Orchestration to leverage Shared Compute to interact with the 3-namespace hierarchy.We have some functions and references to code that are outside of the features ...
- 2104 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod Okay, that makes sense, thank you.What about the approach to identifying these unsupported methods? Is there any documentation of what is unsupported between Unrestricted and Shared?
- 0 kudos
- 2158 Views
- 0 replies
- 0 kudos
duplicate files in bronze delta table
Hello All,I am facing this issue from long time but so far there is no solution. I have delta table. My bronze layer is picking up the old files (mostly 8 days old file) randomly. My source of files is azure blob storage.Those files and not being upd...
- 2158 Views
- 0 replies
- 0 kudos
- 2885 Views
- 2 replies
- 1 kudos
Resolved! Cost of running a job on databricks
Hi All, I need assistance with the cost of running the job on Databricks where I have 20-30 TB (one-time job) and daily data would be around 2GB. The level of transformation would be medium. Source and destination is AWS s3.Looking for your quick re...
- 2885 Views
- 2 replies
- 1 kudos
- 1 kudos
Is there a way to predict the cost before building the solution? I mean we wanted to see our option on different platforms.
- 1 kudos
- 1452 Views
- 1 replies
- 0 kudos
Is it possible to migrate SQL Objects from one workspace to another?
We have SQL Queries and dashboards in workspace dev_01. New workspace dev_02 is created and unity catalog is enabled.I was able to migrate jobs, clusters, DLTs, SQL warehouses, users using APIs. But, while migrating queries using APIs, I can't get th...
- 1452 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm doing something similar, but I haven't run into this parent directory issue. [Actually to be clear I ran into an issue around missing user directories, but I believe that was different than what you describe]. Before migrating the queries, I'm re...
- 0 kudos
| User | Count |
|---|---|
| 1632 | |
| 791 | |
| 513 | |
| 349 | |
| 287 |
