Databricks Community

Hi @ManojkMohan ,To jump in to conversion, is there any particular reason why you don't want to load that csv to Delta format? Delta has multiple advantages over reqular parquet.Things like file skipping, predicate pushdown filtering are much more pe...

Hi @Sainath368 ,I wouldn't use photon for this kind of task. You should use it primarly for ETL transformations where it shines.VACUUM and OPTIMIZE are more of maintenance tasks and using photon would be pricey overkill here.According to documentatio...

Dear @szymon_dybczak and @-werners- ,thank you a lot for for your responses and references! @-werners- , thank you for the link to the announcement article. The availability section lists that "No-Isolation and Unity Catalog Dedicated Clusters" are s...

@nayan_wylde @szymon_dybczak I just tried using a JAR I uploaded to an allowlisted Volume (ojdbc8 of oracle) and I get the same error. it seems like I'm able to install an JAR, but once it's installed my cluster is broken.

Hello @Vamsi_S Good day! Did you Preprocessing Table Names in CI/CD and Generate YAML Dynamically (Recommended for Dynamic, Automated Ingestion)Did you contact your databricks account manager (incase if you working with a company) for feature request...

@shan-databricks there's certainly ways for the schema to evolve within your delta tables that's supported out of the box: https://docs.databricks.com/aws/en/delta/update-schema#enable-schema-evolution To update older records, they'd likely have NULL...

Hello @Phani1 Good day:I have found a whole document on your requirementshttps://community.databricks.com/t5/technical-blog/optimising-data-integration-and-serving-patterns-with-cosmos-db/ba-p/91977 It has a project with it as well. 

I agree.Can you mark your (or someone else´s) answer as solved?  Because I think you won´t be the only one with this issue/feature.

Hi @xavier_db ,Standard access mode has more limitations compared to dedicate access mode. For example, look at the limitations list of standard access mode:Standard compute requirements and limitations | Databricks on AWSNow, compare it to dedicated...

I am running into the same issue. It seems like the `%run` magic command is trying to parse the entire cell content as its arguments. So if you have%run "my_notebook" print("hello")in the same cell, you will get the following error: `Failed to parse...

Core ProblemBronze table is not append-only, but truncate + insert every second.DLT (Delta Live Tables) in continuous mode assumes append-only streaming sources (like Kafka).Because Qlik wipes and replaces data every second, DLT cannot guarantee no d...

Hi @ScottH ,It took me about 2 hours to make it right, but here it is. You need to provide valid date. And you can ask, from where that date is coming from? It's coming from consumer listing: listings = w.consumer_listings.get(id= 'e913bea3-9a37-446c...